Cobrador on Nostr: # Building TollGate Our build pipeline is still undergoing change, but things need to ...
# Building TollGate
Our build pipeline is still undergoing change, but things need to stabilise and become easy for developers (like you) to use. Your feedback and pull requests are most welcome.
### How does OpenWRT manage packages?
OpenWRT uses a package manager called `opkg` to install packages (`.ipk` files), which are compiled to from `Makefiles`. When you build OpenWRT, it clones a [`feeds.conf`](https://github.com/openwrt/openwrt/blob/main/feeds.conf.default) file containing all the required dependencies. You can include a link to your own [custom feed](https://github.com/OpenTollGate/custom-nostr-feed) which contains `Makefiles` that download your repositories of your custom packages into the OpenWRT directory so that they can be build along with the rest of the operating system.
> [!NOTE]
> Beware, compiling with the full buildroot does everything from scratch, every time you want to try your custom program. You may want to avoid this when making frequent changes to the packages in your custom feed.
>
> This takes too long, (almost) no-one has that much time for freedom..
### Using the SDK
Fortunately, there is something called an SDK, which is essentially a simplified buildroot containing a stripped down version of the OpenWRT source tree. The SDK contains:
• Compiler toolchains and libraries (precompiled)
• Scripts and makefiles to handle package configuration and compilation
Since the compiler and most dependencies are already built when you use the SDK, you only compile your custom feeds or packages. Now we can just [download & unpack](https://github.com/OpenTollGate/tollgate-sdk/blob/71e37959c3e0d4a4ef9c8d053f9d9fdb72a797cb/build-firmware#L141-L148) the SDK, [place our custom feeds](https://github.com/OpenTollGate/tollgate-sdk/blob/71e37959c3e0d4a4ef9c8d053f9d9fdb72a797cb/build-firmware#L156-L162), use `make defconfig` to create a configuration file and `make package/[our_package]/compile` to compile the individual packages.
Once we have our packages, we upload them to blossom servers and create a nostr event listing the binaries we uploaded, the servers that they can be found on and the commit hashes of the repositories that the packages were compiled from.
Now that we have our packages on blossom servers, routers can just listen for signed nostr events announcing compatible updates, download the packages and install them using the OpenWRT package manager. Disclaimer: we need you to make this client.
> [!Your Turn]
> [Here](
### Using the image-builder
Another option is to use the image-builder to download a pre-defined list of packages and [install them](https://github.com/OpenTollGate/tollgate-image-builder/blob/d4d48ea0df756b3c25aeb0b77201c242a1a789aa/build-firmware#L257-L268) in an OpenWRT image. We can also place configuration files that we need in the image's filesystem, which could be more reliable (declarative) than modifying configuration files using the install section of a package's `Makefile`.
Now we have an OpenWRT image that we can copy to the router using `scp` and install with `sysupgrade -n [image_name].bin`.
> [!Your Turn]
> [Here](
### Outlook - runners in gitworkshop.dev
We want to create git actions for the tool that compiles packages with the SDK and the tool that installs these packages into an OpenWRT image so that we can tap into stranded compute in order to iterate more quickly.
Fortunately there is this awesome tool called `ngit`, which creates nostr events for our `git` commits and enables us to use nostr to create `issues` and `pull requests`. Fortunately arjenstens (nprofile…7u3d) is integrating an open-source runner for git actions with gitworkshop so that `npubs` can stack SATs by running your git actions for you.
### Something is missing, wrong or outdated?
Lets fix it. Make a pull request, DM the maintainers, edit the event. We need you!
Thanks to #SovEng for creating the conditions for this to happen. Please [join us](https://sovereignengineering.typeform.com/SEC-04?typeform-source=sovereignengineering.io) in Madeira if you want to help.
#TollGate
Our build pipeline is still undergoing change, but things need to stabilise and become easy for developers (like you) to use. Your feedback and pull requests are most welcome.
### How does OpenWRT manage packages?
OpenWRT uses a package manager called `opkg` to install packages (`.ipk` files), which are compiled to from `Makefiles`. When you build OpenWRT, it clones a [`feeds.conf`](https://github.com/openwrt/openwrt/blob/main/feeds.conf.default) file containing all the required dependencies. You can include a link to your own [custom feed](https://github.com/OpenTollGate/custom-nostr-feed) which contains `Makefiles` that download your repositories of your custom packages into the OpenWRT directory so that they can be build along with the rest of the operating system.
> [!NOTE]
> Beware, compiling with the full buildroot does everything from scratch, every time you want to try your custom program. You may want to avoid this when making frequent changes to the packages in your custom feed.
>
> This takes too long, (almost) no-one has that much time for freedom..
### Using the SDK
Fortunately, there is something called an SDK, which is essentially a simplified buildroot containing a stripped down version of the OpenWRT source tree. The SDK contains:
• Compiler toolchains and libraries (precompiled)
• Scripts and makefiles to handle package configuration and compilation
Since the compiler and most dependencies are already built when you use the SDK, you only compile your custom feeds or packages. Now we can just [download & unpack](https://github.com/OpenTollGate/tollgate-sdk/blob/71e37959c3e0d4a4ef9c8d053f9d9fdb72a797cb/build-firmware#L141-L148) the SDK, [place our custom feeds](https://github.com/OpenTollGate/tollgate-sdk/blob/71e37959c3e0d4a4ef9c8d053f9d9fdb72a797cb/build-firmware#L156-L162), use `make defconfig` to create a configuration file and `make package/[our_package]/compile` to compile the individual packages.
Once we have our packages, we upload them to blossom servers and create a nostr event listing the binaries we uploaded, the servers that they can be found on and the commit hashes of the repositories that the packages were compiled from.
Now that we have our packages on blossom servers, routers can just listen for signed nostr events announcing compatible updates, download the packages and install them using the OpenWRT package manager. Disclaimer: we need you to make this client.
> [!Your Turn]
> [Here](
) are some practical details for those who actually want to do it.quoting nevent1q…s0pk### Using the SDK in practice
Earlier we explained the ins and outs of building packages and images for OpenWRT. Now lets actually build a package rather than just talking about it.
##### Lets clone the SDK
```
git clone https://github.com/OpenTollGate/tollgate-sdk.git
```
##### Lets install a tool to interact with blossom servers
```
./blossom-installer.sh
```
##### Lets install a tool to interact with nostr relays
```
./noscl-installer.sh
```
##### Lets prepare a file containing the keys for these tools
```
cat blossom_secrets.json
{
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
],
"relays": [
"wss://orangesync.tech",
"wss://nostr.mom",
"wss://nostr.chaima.info"
],
"secret_key": "nsec[your_secret_key]",
"secret_key_hex": "[hex_version_of_your_secret_key]",
"public_key": "npub[your_public_key]",
"public_key_hex": "[hex_version_of_your_public_key]"
}
```
##### Lets go!
```
./build-firmware gl-mt3000
```
The above script
* installs some dependencies
* downloads the SDK to `/tmp/openwrt-sdk/openwrt-sdk-23.05.3-mediatek-filogic_gcc-12.3.0_musl.Linux-x86_64`
* compiles `golang` as part of a custom feed because the newest version of `golang` is too old for some of [our dependencies](https://github.com/OpenTollGate/tollgate-module-relay-go/blob/main/src/go.mod)
* compiles the remaining custom feeds
> [!Patience]
> Even though we are using the SDK, the above steps will take a while if your running them for the first time. This might be your chance to get a coffee.
##### Are you winning?
Eventually you should see the following output. This is the nostr event that your npub just broadcasted to announce that you compiled your custom feed successfully and that your packages are available on this listed blossom servers. Maybe franzap (nprofile…ucf3) has advice on how we can improve this workflow.
```
{
"binaries": {
"tollgate-module-whoami-go_0.1-1_aarch64_cortex-a53.ipk": {
"file_hash": "cc84a9cc453f5c6a5ff37c8162f5161178ecd7027f29530b5ffc955fd1a7d196",
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
]
},
"golang-src_1.23.4-1_aarch64_cortex-a53.ipk": {
"file_hash": "db629bae7a6ab300ee34032cb11717447ffdbfc25260fed8554c991722d29ee8",
"servers": []
},
"golang_1.23.4-1_aarch64_cortex-a53.ipk": {
"file_hash": "95d1e430d654d99c1ca041dfbabc535a1fb72e6d0a19f960f731cc2ab612ccb6",
"servers": []
},
"tollgate-module-crowsnest-go_0.1-1_aarch64_cortex-a53.ipk": {
"file_hash": "3200a4f565886798583403e9d43d515f303172a697e31b55e70d63a7536d6fc9",
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
]
},
"tollgate-module-merchant-go_0.1-1_aarch64_cortex-a53.ipk": {
"file_hash": "85e98940f78e3ba1088692a4e812759ba7bfa997af3552acb61c31ac989a6995",
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
]
},
"tollgate-module-relay-go_0.1-1_aarch64_cortex-a53.ipk": {
"file_hash": "403e81351cb019df61e96e62fe88c4d7f9783fb5db9e6ffe59ff0d224e1657bc",
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
]
},
"golang-doc_1.23.4-1_aarch64_cortex-a53.ipk": {
"file_hash": "6589935e4c472e97280f18e795a6f12292490ee0149ea307556af475548ae36f",
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
]
},
"tollgate-module-valve-go_0.1-1_aarch64_cortex-a53.ipk": {
"file_hash": "9fab480f503aef59621db43aa69be671f9d78139c7cb79853279d3d46510e3bc",
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
]
}
},
"target_info": {
"target_platform": "mediatek-filogic",
"full_arch": "aarch64_cortex-a53"
},
"feed_info": {
"71e37959c3e0d4a4ef9c8d053f9d9fdb72a797cb": {
"directory": "toll_gate_sdk",
"branch": "main"
},
"278cf417b3b27298b8b1ea9fc116177dd227eb71": {
"directory": "custom",
"branch": "main"
},
"1292a493a68e1ef144429ac33848eabf60a9b59c": {
"directory": "base",
"branch": "openwrt-23.05"
}
}
}
```
##### How can you help? #help
We have plenty of questions that could benefit from your expertise:
* how can we make our custom packages smaller - static/dynamic linking?
* how can we configure [`golang.mk`](https://primal.net/e/nevent1qqs95086an45xa94kxjml23t3774xr2kr295pw0m4u4d4jyaazfcfrsserkkj) so that we can target pretty any OpenWRT router successfully?
* how can we get the builds to run more quickly?
* does it make sense to create a separate git action for each of our modules or is the `set-up` and `tear-down` too expensive?
### Using the image-builder
Another option is to use the image-builder to download a pre-defined list of packages and [install them](https://github.com/OpenTollGate/tollgate-image-builder/blob/d4d48ea0df756b3c25aeb0b77201c242a1a789aa/build-firmware#L257-L268) in an OpenWRT image. We can also place configuration files that we need in the image's filesystem, which could be more reliable (declarative) than modifying configuration files using the install section of a package's `Makefile`.
Now we have an OpenWRT image that we can copy to the router using `scp` and install with `sysupgrade -n [image_name].bin`.
> [!Your Turn]
> [Here](
) are some practical details for those who actually want to do it.quoting nevent1q…6cjn### Using the image-builder in practice
Earlier we explained the ins and outs of building packages and images for OpenWRT. Now lets actually build a package rather than just talking about it.
##### Lets clone the image-builder
```
git clone https://github.com/OpenTollGate/tollgate-image-builder.git
```
##### Lets install a tool to interact with blossom servers
```
curl -sSL https://raw.githubusercontent.com/OpenTollGate/tollgate-sdk/refs/heads/main/blossom-installer.sh | sudo bash
```
##### Lets install a python library for interacting with nostr relays
```
pip3 install nostr
```
##### Lets prepare a file containing the keys for these tools
```
cat blossom_secrets.json
{
"servers": [
"https://files.v0l.io/",
"https://nostr.download/",
"https://blossom.poster.place/"
],
"relays": [
"wss://orangesync.tech",
"wss://nostr.mom",
"wss://nostr.chaima.info"
],
"secret_key": "nsec[your_secret_key]",
"secret_key_hex": "[hex_version_of_your_secret_key]",
"public_key": "npub[your_public_key]",
"public_key_hex": "[hex_version_of_your_public_key]"
}
```
##### Lets go!
```
./build-firmware gl-mt3000
```
The above script
* downloads the OpenWRT image-builder
* downloads your [custom packages](https://github.com/OpenTollGate/tollgate-image-builder/blob/d4d48ea0df756b3c25aeb0b77201c242a1a789aa/build-firmware#L244-L246) from blossom
* [installs all the packages](https://github.com/OpenTollGate/tollgate-image-builder/blob/d4d48ea0df756b3c25aeb0b77201c242a1a789aa/build-firmware#L267-L275) in an OpenWRT image
> [!Known bug]
> You might get an error saying that the websocket connection to your nostr relays can't be opened. In that case, just abort and run the above command again. It often works on the second try.
> ```
> DEBUG: Error getting event: socket is already closed.
No valid events found for architecture: aarch64_cortex-a53
> ```
> Pull requests welcome..
##### Are you winning?
Eventually you should see the following output. This is the nostr event that your npub just broadcasted to announce that you compiled your custom feed successfully and that your packages are available on this listed blossom servers. Maybe franzap (nprofile…ucf3) has advice on how we can improve this workflow.
```
Number of ids (unique uids + gids) 1
Number of uids 1
unknown (0)
Number of gids 1
unknown (0)
Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
compressed data, compressed metadata, compressed fragments,
no xattrs, compressed ids
duplicates are removed
Filesystem size 64302.32 Kbytes (62.80 Mbytes)
24.24% of uncompressed filesystem size (265222.92 Kbytes)
Inode table size 109854 bytes (107.28 Kbytes)
20.87% of uncompressed inode table size (526354 bytes)
Directory table size 140778 bytes (137.48 Kbytes)
38.40% of uncompressed directory table size (366654 bytes)
Number of duplicate files found 494
Number of inodes 16283
Number of files 14440
Number of fragments 371
Number of symbolic links 242
Number of device nodes 1
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 1600
Number of hard-links 0
Number of ids (unique uids + gids) 1
Number of uids 1
unknown (0)
Number of gids 1
unknown (0)
64302+1 records in
64303+0 records out
65846272 bytes (66 MB, 63 MiB) copied, 0.091405 s, 720 MB/s
64302+1 records in
64303+0 records out
65846272 bytes (66 MB, 63 MiB) copied, 0.091405 s, 720 MB/s
/tmp/openwrt-build/openwrt-imagebuilder-23.05.3-mediatek-filogic.Linux-x86_64/staging_dir/host/bin/tar: Option --mtime: Treating date '@1711145382' as 2024-03-22 23:09:42
sysupgrade-glinet_gl-mt3000/
sysupgrade-glinet_gl-mt3000/CONTROL
sysupgrade-glinet_gl-mt3000/kernel
/tmp/openwrt-build/openwrt-imagebuilder-23.05.3-mediatek-filogic.Linux-x86_64/staging_dir/host/bin/tar: Option --mtime: Treating date '@1711145382' as 2024-03-22 23:09:42
sysupgrade-glinet_gl-mt3000/
sysupgrade-glinet_gl-mt3000/CONTROL
sysupgrade-glinet_gl-mt3000/kernel
sysupgrade-glinet_gl-mt3000/root
sysupgrade-glinet_gl-mt3000/root
Calculating checksums...
Calculating checksums...
Build successful! Firmware image created at:
Image size (62.7952
62.7952MB)
/tmp/openwrt-build/openwrt-imagebuilder-23.05.3-mediatek-filogic.Linux-x86_64/bin/targets/mediatek/filogic/openwrt-23.05.3-mediatek-filogic-glinet_gl-mt3000-squashfs-sysupgrade.bin
```
##### How do I install this thing?
Copy the `sysupgrade.bin` file to the `/tmp` directory of your router.
```
scp /tmp/openwrt-build/openwrt-imagebuilder-23.05.3-mediatek-filogic.Linux-x86_64/bin/targets/mediatek/filogic/openwrt-23.05.3-mediatek-filogic-glinet_gl-mt3000-squashfs-sysupgrade.bin root@[routers-ip-address]:/tmp
```
> [!NOTE]
> The `/tmp` directory is part of your device's RAM rather than its (small) flash storage, so it should have a lot of space for such binaries.
Now lets install it
```
sysupgrade -n [your-sysupgrade-file].bin
```
Please leave your router plugged in and give it a minute or two of time to install the new image. It might show up with a new IP address and you will be prompted to delete the old router from your cached `ssh` keys when you try to log in to it.
##### How can you help? #help
* do you know how to make git actions for this?
* we are [struggling](nevent1q…pezm) to build images for the `gl-ar300m` target. This might be connected with the fact that there are two similar devices (`gl-ar300m` and `gl-ar300m16`) that don't have the exact same architecture
* have you fixed that bug already?
* please DM us if you have questions or your looking for other ways to contribute
### Outlook - runners in gitworkshop.dev
We want to create git actions for the tool that compiles packages with the SDK and the tool that installs these packages into an OpenWRT image so that we can tap into stranded compute in order to iterate more quickly.
Fortunately there is this awesome tool called `ngit`, which creates nostr events for our `git` commits and enables us to use nostr to create `issues` and `pull requests`. Fortunately arjenstens (nprofile…7u3d) is integrating an open-source runner for git actions with gitworkshop so that `npubs` can stack SATs by running your git actions for you.
### Something is missing, wrong or outdated?
Lets fix it. Make a pull request, DM the maintainers, edit the event. We need you!
Thanks to #SovEng for creating the conditions for this to happen. Please [join us](https://sovereignengineering.typeform.com/SEC-04?typeform-source=sovereignengineering.io) in Madeira if you want to help.
#TollGate