Note: This project was originally released as Clawdbot, then renamed to Moltbot, and is now called OpenClaw. If you followed this guide under one of the previous names, the steps are the same — only the package and command names have changed. This guide has been updated to reflect the current name.

OpenClaw seems to be the biggest hype in January 2026. Many people talk about it, the amount of development poured into it and the speed it’s developing is crazy.

The documentation is thorough, but even with Claude Code I didn’t come to a working solution without manually intervening.

So what follows is a quick walkthrough on how to get OpenClaw working on a fresh Ubuntu 24.04 VPS from Hetzner.

#1. Create VPS

Log into your cloud provider and create a simple VPS. 2 vCPUs and 4GB of RAM is more than enough to start. I used the cheapest Hetzner option:

#2. Basic VPS setup

We’ll start with logging into the server, updating, and creating a dedicated user to use with OpenClaw.

1ssh root@your-server-ip # enter the password you received via email
2 
3apt update && apt upgrade -y # update packages
4# if prompted to reboot, do it.
5 
6adduser openclaw # create the new user. You can leave all details empty, but don't forget to set a strong password
7usermod -aG sudo openclaw # add the user to the sudoers group so we can use `sudo`
8 
9su - openclaw # change to the new user

#3. Securing your VPS

Before installing OpenClaw, we should secure the server with some basic security measures. These steps will protect your VPS from common attacks.

#Setup SSH key authentication

First, let’s set up SSH key authentication so you can log in without a password. Run this on your local machine:

1# Generate SSH key if you don't have one already
2ssh-keygen -t ed25519 -C "your_email@example.com"
3 
4# Copy your public key to the server
5ssh-copy-id openclaw@your-server-ip

Test the SSH key login by opening a new terminal and connecting:

1ssh openclaw@your-server-ip

If you can log in without entering a password, the key authentication is working.

#Disable password authentication

Now that key-based authentication is working, let’s disable password login. This prevents brute force attacks.

1# Edit SSH configuration
2sudo vim /etc/ssh/sshd_config

Find and update these lines (uncomment if needed):

1PasswordAuthentication no
2PubkeyAuthentication yes
3PermitRootLogin no

Save the file (:wq) and restart SSH:

1sudo systemctl restart ssh

#Install fail2ban

Fail2ban monitors login attempts and automatically blocks IP addresses that show malicious behavior.

1sudo apt install fail2ban -y
2 
3# Create a local configuration file
4sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
5 
6# Edit the configuration
7sudo vim /etc/fail2ban/jail.local

Find the [sshd] section and ensure it looks like this:

1[sshd]
2enabled = true
3port = ssh
4logpath = /var/log/auth.log
5maxretry = 5
6bantime = 3600

Start and enable fail2ban:

1sudo systemctl start fail2ban
2sudo systemctl enable fail2ban
3 
4# Check status
5sudo fail2ban-client status sshd

#Setup UFW firewall

Configure a basic firewall to only allow necessary connections:

 1# Install UFW
 2sudo apt install ufw -y
 3 
 4# Allow SSH (important - do this first!)
 5sudo ufw allow ssh
 6 
 7# Enable the firewall
 8sudo ufw enable
 9 
10# Check status
11sudo ufw status

If you plan to expose OpenClaw through a web interface later, you can allow HTTP/HTTPS:

1sudo ufw allow http
2sudo ufw allow https

#4. Installing prerequisites

We need some packages for OpenClaw to work properly. Verify that you are logged in as the openclaw user or you might run into permission problems.

#Node

In my testing I found npm to be the most reliable package manager, but you can also try pnpm. You can find the up-to-date install instructions at nodejs.org/en/download.

 1# Install Node
 2# Download and install nvm:
 3curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
 4 
 5# in lieu of restarting the shell
 6\. "$HOME/.nvm/nvm.sh"
 7 
 8# Download and install Node.js:
 9nvm install 24
10 
11# Verify the Node.js version:
12node -v # Should print "v24.13.0".
13 
14# Verify npm version:
15npm -v # Should print "11.6.2".

#Homebrew

Homebrew is a package manager. It is needed by some skills (instructions used by OpenClaw to enhance its abilities).

1/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

#Tailscale (optional)

Tailscale is a Zero Trust identity-based connectivity platform. It can be used to securely connect your local computer with your OpenClaw gateway making it possible for the gateway to do things on your local computer. I currently don’t have a use case for this, so you might also skip it.

1curl -fsSL https://tailscale.com/install.sh | sh
2sudo tailscale up

#5. OpenClaw

Now the fun begins 🦞. Let’s install OpenClaw and onboard you.

#Installation

1npm i -g openclaw

This might take a while, but that’s it. You’ve installed OpenClaw and are ready to onboard.

#Onboarding

Run openclaw onboard. It will take you through its interactive onboarding guide. Below are the settings I used but be aware that OpenClaw is very actively developed, so settings might change quickly. I’ve left some comments.

 1◆  I understand this is powerful and inherently risky. Continue?
 2│  Yes
 3|
 4◆  Onboarding mode
 5│  ○ QuickStart
 6│  ● Manual (Configure port, network, Tailscale, and auth options.)
 7|  # Use manual mode for more control
 8|
 9◆  What do you want to set up?
10│  ● Local gateway (this machine) (Gateway reachable (ws://127.0.0.1:18789))
11│  ○ Remote gateway (info-only)
12|  # It might seem counterintuitive at first, but you want the Gateway to run locally on the VPS
13|
14◆  Workspace directory
15│  /home/openclaw/clawd
16|
17◆  Model/auth provider
18│  ● OpenAI (Codex OAuth + API key)
19│  ○ Anthropic
20│  ○ MiniMax
21│  ○ Qwen
22│  ○ Synthetic
23│  ○ Google
24│  ○ Copilot
25|  ...
26|  # Choose what's right for you! I've used my Claude Code subscription to authenticate. Refer to the documentation if needed.
27|
28◆  Gateway port
29│  18789
30|
31◆  Gateway bind
32│  ● Loopback (127.0.0.1)
33│  ○ LAN (0.0.0.0)
34│  ○ Tailnet (Tailscale IP)
35│  ○ Auto (Loopback → LAN)
36│  ○ Custom IP
37|
38◆  Gateway auth
39│  ○ Off (loopback only)
40│  ● Token (Recommended default (local + remote))
41│  ○ Password
42|
43|  # this will give a token you may use to access the dashboard
44◆  Tailscale exposure
45│  ○ Off
46│  ● Serve (Private HTTPS for your tailnet (devices on Tailscale))
47│  ○ Funnel
48|  # choose "off" if you don't use Tailscale
49|
50◆  Reset Tailscale serve/funnel on exit?
51│  ○ Yes / ● No
52|  # choose "no" if you used "Serve" as Tailscale exposure
53|
54◆  Configure chat channels now?
55│  ● Yes / ○ No
56|  # You probably want to do this. Having access via chat makes the setup much more flexible.
57|  # Choose your favorite provider. When in doubt just use Telegram, they have the easiest setup for a chat bot.
58│
59◇  Skills status ────────────╮
60│                            │
61│  Eligible: 13              │
62│  Missing requirements: 38  │
63│  Blocked by allowlist: 0   │
64│                            │
65├────────────────────────────╯
66│
67◆  Configure skills now? (recommended)
68│  ● Yes / ○ No
69|  # Choose `Yes` and select all Skills you want to have installed with `Spacebar` and confirm with `Enter`
70|  # If unsure just skip it for now, you can always configure this later
71│
72◆  Preferred node manager for skill installs
73│  ● npm
74│  ○ pnpm
75│  ○ bun
76│
77◆  Set GOOGLE_PLACES_API_KEY for goplaces?
78│  ○ Yes / ● No
79|  # Unless you have API keys just answer no to all the API key questions
80│
81◆  Enable hooks?
82│  ◼ Skip for now
83│  ◻ 🚀 boot-md
84│  ◻ 📝 command-logger
85│  ◻ 💾 session-memory
86|  # Skip, unless you know what you are doing (or read the manual)
87│
88◆  Install Gateway service (recommended)
89│  ● Yes / ○ No
90│
91◆  Gateway service runtime
92│  ● Node (recommended) (Required for WhatsApp + Telegram. Bun can corrupt memory
93on reconnect.)
94│
95◆  How do you want to hatch your bot?
96│  ● Hatch in TUI (recommended)
97│  ○ Open the Web UI
98│  ○ Do this later
99|  # After confirming this you will be put into a TUI (Terminal User Interface) to "hatch" your new assistant. Have fun!

#Telegram

After the gateway has started (the onboarding setup should have shown you a success message) you can message your bot if you created one with /start to pair your Telegram chat sessions to your OpenClaw gateway. Everything is explained in the chat.

#6. Use cases and configuration

You might be tempted to configure workflows, automations, or other features via the Dashboard UI. While this works for some things, I highly recommend using the chat interface instead.

Simply describe what you want to achieve. OpenClaw is remarkably good at understanding intent and will guide you through the setup process. For example, I’ve long wanted a summary of my bookmarked tweets. I tend to hit the bookmark icon and then forget to review them later. I asked OpenClaw to send me a daily digest, and it walked me through the entire process: setting up bird (the CLI for interacting with X), configuring the skill, and even sending a test message to confirm everything worked.

If there’s something you want to accomplish, just ask. It genuinely feels like having a personal assistant. Rather than responding with “I’m sorry, I can’t do that,” it proposes a plan to achieve your desired outcome.

The beauty of this solution is its simplicity: the Kindle only connects to Wi-Fi when you turn it on, making it a perfect trigger for automation.

#How it works

When you activate your Kindle, it connects to your Wi-Fi network. Home Assistant detects this through a device tracker and starts a 10-minute timer. After the timer expires, your bedroom lights gradually fade out.

I added a time condition to prevent the lights from turning off too early in the evening - nobody wants their lights to go out at 7 PM just because they picked up their Kindle.

#Setting up the automation

Create an automation with the following configuration:

 1alias: "Bedroom: Turn off night lamp after 10 minutes of reading"
 2description: ""
 3mode: single
 4triggers:
 5  - entity_id:
 6      - device_tracker.kindle
 7    to: home
 8    trigger: state
 9conditions:
10  - condition: time
11    before: "23:59:58"
12    after: "21:00:01"
13actions:
14  - delay:
15      hours: 0
16      minutes: 10
17      seconds: 0
18      milliseconds: 0
19  - data:
20      transition: 5
21    target:
22      area_id: bedroom
23    action: light.turn_off

Important: Replace bedroom with your bedroom area ID, or use specific entity IDs if you haven’t configured areas in Home Assistant.

The transition: 5 parameter gradually dims the lights over 5 seconds, preventing the jarring experience of sudden darkness when you’re likely already half asleep.

#Setting up device tracking for your Kindle

If your Kindle doesn’t appear as a device tracker yet, you’ll need to add it:

1. Open Settings → Devices & Services
2. Check if your router integration is set up (most routers are supported)
3. Your Kindle should appear as a device once it connects to Wi-Fi
4. Note the entity ID (likely something like device_tracker.kindle)

That’s it! A simple automation that makes bedtime just a little bit better.

Last time I’ve explained my motiviation behind building a project like this, what features I’ve built (like a notification system for the digital timetable of my kid) and what my plans were for the future.

Over half a year later and I still use many of the things I’ve built, discarded some, and removed things that didn’t work.

#What worked

All features I’ve implemented initially (digital timetable with notifications, vocabulary tests, goal tracking, bookmarks) are still in place.
Although recently there weren’t many vocabulary tests done (learning methods changed a bit, grammar coming in).

But: I don’t use an LLM anymore to create motivational messages for goal tracking. Initially, the thought of having a “personal assistant” sounded great, but after implementation it was just plain boring. That might be a skill issue on my part, but I’ve removed it completely for now.

The goal tracking itself still motivates me to stay on track. To make it more rewarding I’ve created an XP system which rewards streaks.
Although I must say this isn’t motivating me as much as I’ve hoped, so I might remove it in the near future.

That’s one of the aspects of this project that I like most: it feels like a living organism adjusting to my needs. Maybe this also feeds a hidden god-complex, I don’t know.

#What didn’t work

In the closing words of my last post I had the plan to build a daily digest, which I did.
But turns out, this is also not as useful as I hoped, so I quickly removed it again. The problem I’ve faced was that in regard to fulfil the requirement of the aforementioned personal assistant, the timing would need to be much better. I don’t want to see nearly identical messages each day at 9 am, but when I’m ready to consume it. Maybe I need a Home Assistant integration to make this better.

#What is new

I’ve added a bunch of new stuff since February.

#Gym tracker

Before, I was using the app my gym provided to track my sessions. I also used Alpha Progression for maybe a year. The gym provided system wasn’t very user friendly and I didn’t like that some simple functions in Alpha Progression needed a subscription.

So like any good programmer I underestimated the complexity massively and built my own gym session tracker.

I started pretty small. Just having training plans, exercises and a way to track each set. One thing important for me was to have a clear indicator when to progress. It’s a simple thing but helps me get better. Alpha Progress called this “smart progression hints” or something like that. Nothing about this has to be smart. If I’ve hit the upper bound of my rep range I put on more weight. Easy as that.

#Packlist Generator

I grep up in a household where we had a 4 pages long Excel sheet to plan what to take with us on vacation. This causes irreversible brain damage to a little human being, causing them to write a program they can use to track what to pack for a journey.

The interesting part wasn’t the items itself but the business logic what should be packed. Each journey can have multiple properties (like traveling-with-kid, business-trip) which is be used to match to specific items.

Each property has a matching function defining if the item should be included in the journey. For properties like warm-weather or rain-expected I needed a Weather API implementation. But most APIs only give you a forecast of around 14 days. While this might sound like there are APIs with a more extensive forecast that’s unfortunately not the case. The most limiting factor is the weather models, getting less and less precise farther into the future.

But I didn’t want to plan journeys only 2 weeks ahead. While this would be sufficient in most cases, solving the problem of looking farther into the future was the problem I wanted to solve.

I’m using Open-Meteo to get the weather. They have great APIs, good documentation and on top of that are free.

If the journey is planned more days ahead I’m using an average from the past five years to assume what the weather might me.

 1/**
 2 * @param  array{latitude:float,longitude:float}  $coordinates
 3 * @return Collection<int, DayForecast>
 4 */
 5private static function fetchHistoricalAverageForecast(array $coordinates, CarbonImmutable $start, CarbonImmutable $end): Collection
 6{
 7    $historicalStartDate = $start->subYears(self::HISTORICAL_YEARS_TO_AVERAGE);
 8    $historicalEndDate = $end->subYears(1);
 9 
10    $params = [
11        'latitude' => $coordinates['latitude'],
12        'longitude' => $coordinates['longitude'],
13        'daily' => 'weather_code,temperature_2m_max,temperature_2m_min,precipitation_sum',
14        'start_date' => $historicalStartDate->format('Y-m-d'),
15        'end_date' => $historicalEndDate->format('Y-m-d'),
16    ];
17 
18    $response = self::makeApiCall(self::HISTORICAL_API_URL, $params);
19 
20    if (! $response || ! isset($response['daily'])) {
21        return collect();
22    }
23 
24    return self::processHistoricalData($response['daily'], $start, $end);
25}

No idea yet how precise this will be, but for a rough estimate it should suffice. I always have the possibility to recreate the list if needed.

#LastFM importer

Over the last 18 years I’ve accumulated over 215k scrobbles on Last.fm. Apart from maybe Google this is one of the services I use the longest and without much interruptions.

In the last months I more and more lean towards the idea of owning my own data, so I wrote an importer for Last.fm.

Because of the way the Last.fm API works there unfortunately is no easy way to export everything.
Pages are counted from present to past and there is no way to reverse this logic. This could mean that a scrobble will change the currently first page the second.

But I think the solution I came up with is simple and elegant:

 1public function handle(LastFm $lastFm): void
 2{
 3    $localScrobbles = Scrobble::query()->count();
 4 
 5    $latestScrobble = $lastFm->getRecentTracks(1);
 6 
 7    // Get the latest scrobble to get the total scrobbles from the API response
 8    $totalScrobbles = (int) data_get($latestScrobble, '@attr.total');
 9 
10    // If we have the same amount of scrobbles locally, bail
11    if ($totalScrobbles === $localScrobbles) {
12        return;
13    }
14 
15    // calculate the total number of pages
16    $totalPages = (int) ceil($totalScrobbles / self::LIMIT);
17 
18    // calculate the difference to know which page to get
19    $pageToFetch = max($this->page ?? (int) ceil($totalPages - ($localScrobbles / self::LIMIT)), 1);
20 
21    // get the scrobbles and reject all with the exact same date
22    $allScrobbles = collect(data_get($lastFm->getRecentTracks(limit: self::LIMIT, page: $pageToFetch), 'track'));
23    $scrobbles = $allScrobbles->reject(fn ($scrobble) => ! data_get($scrobble, 'date.uts'));
24 
25    // format and save to db
26    $data = $scrobbles->map(function ($scrobble) {
27        return [
28            'artist' => data_get($scrobble, 'artist.#text'),
29            'album' => data_get($scrobble, 'album.#text'),
30            'track' => data_get($scrobble, 'name'),
31            'played_at' => Carbon::createFromTimestamp(data_get($scrobble, 'date.uts')),
32            'payload' => json_encode($scrobble),
33        ];
34    })->filter();
35 
36    Scrobble::query()->upsert($data->toArray(), ['artist', 'track', 'played_at']);
37 
38    // if we are on the first page, bail
39    if ($pageToFetch === 1) {
40        return;
41    }
42 
43    // recursive logic, if we had Scrobbles get the next page
44    // if not, fetch the same page to make sure we now have all Scrobbles
45    self::dispatch(($scrobbles->count() ? $pageToFetch - 1 : $pageToFetch));
46}

#Todo Management

It started like a simple todo list tutorial. But I’ve had some ideas and concepts in mind that manifested over the years of usage of other tools.

• There are no overdue tasks. A task is either due today, or not. I can’t do something yesterday so don’t punish me for not meeting a deadline but let it gracefully roll over.
• Infinite sub tasks. It’s known that big tasks fuel procrastination because you don’t know where to start. So I wanted a system that is able to handle a task no matter how big, by giving me the possibility to break it down into as small sub tasks as needed.
• Notes integration. The border between notes and tasks is sometimes very thin. Oftentimes a finished task becomes a note or some task spawns from a note. So every task in my system can be a note or a task and it can be switched around as needed. Apart from completing they have the same parts:
  • description
  • links
  • attachments
  • comments
  • subtasks / subnotes

The first version in April 2025 didn’t have that much functionality. It was a very basic todo list with only the notes added.
But it evolved and evolved as I saw what worked and what didn’t work. A few months in I moved all my notes from Obsidian over and haven’t looked back since.

Some time in the future I want to tackle the personal assistant again, using all my notes and tasks as context.

#Printer System using a receipt printer

This one is the most controversial project I’ve done - at least when talking about it in my social circle. While nobody ever really sees most of the things I’m programming: the receipt printer on my desk will rise questions.

I’ve read this great post by Laurie: A receipt printer cured my procrastination. It resonated so much with how I see task management, e.g. he also advocates for breaking big tasks down into smaller ones.
So I’ve started researching receipt printers, how they can be connected and how I could print things on it.

What a fun rabbit hole! Building stuff on the web fascinates me since nearly 20 years. But building stuff that interacts with the physical world had some kind of magic feeling. I can now press a little print button in my task management and it prints the task nearly instantly. Regardless of where I am.

How I’ve built the necessary parts will be a post of its own. But let me tell you: having a physical representation of a task helped me very much staying on top of my todos. More like any other system before. I highly recommend it - if you are able to live with the snarky comments you most likely will get.

The term “Life OS” is often associated with a note taking system helping you manage your life. It does so by giving you structures to organize your thoughts, tasks, and goals.

For me, that’s not enough. An Operating System should be able to manage your life, not just help you manage it. It should be able to automate tasks, remind you of important things and be truly made for you.

In my opinion a note taking system falls short of that. It’s a great tool to help you manage your thoughts, but it’s not a Life OS.

#Motivation behind building a Life OS

For multiple months I’ve juggled with the thought of creating a system that is able to combine all the different digital sources that are part of my life.

The tipping point for me was when my kid got into a new school and got a digital school schedule, namely Webuntis. It contains information about the classes, homework and upcoming tests. The concept is great but the UX is, sorry Untis-Team, terrible.

• There are no notifications
• You don’t see all homeworks or tests at a glance but have to check each day individually
• abbreviations are used that are not explained anywhere

But: they have an API*. So I thought: why not build a system that combines all the different sources of information that are part of my life? I love working with APIs or with data from different sources that do things.

So I started building a Life OS.

* The API sucks, also. But that’s a different story.

#What’s inside

I’m using the TALL stack, which stands for Tailwind CSS, Alpine.js, Laravel and Livewire. I’ve been using the stack professionally for some time now, so I’m very comfortable with it.

For content management and blogging in markdown I’m using Prezet. It’s a package that allows for easy markdown blogging in Laravel. Since all the articles I’ve written before are in markdown it was a great fit and made the switch even easier.

#Features

To this day I’ve implemented the following features:

Webuntis Integration

• it uses the API to fetch all the needed data like the classes, homeworks, tests and news
• displays the data in a nice way
• sends me a Discord notification if a class was cancelled or a new test was added or new news was published

Vocabulary Tests

This is for my kid. And a little bit for me, too. To create a good vocabulary learning system takes a lot of time if you do it analog. So I created a system that allows me to create vocabulary tests and my kid to learn them.

• I can add all current words with their translations
• I can create tests with a certain amount of words
• The system keeps track of the score by subtracting wrong from right answers, therefore allowing us to focus on words that are harder to remember

Goal Tracking

Although there are so many goal tracking apps out there I was never quite satisfied with some part of them. Either the notifications where bothering me, or the UX was not good enough, or the features were too much or too little.

So I created a goal tracking system that allows me to create goals, track them and get notifications if I haven’t worked on them for a certain amount of time.

I’m using LLMs for creating the messages, incorporating the weather so it feels more like a personal coach instead of a dumb notification.

Next steps are the implementation of streaks to motivate me even more.

Bookmarks

I’m not the most organized person when it comes to bookmarks. Most of the time I just keep some tabs open, never read them, close them and when I need them can’t remember where I’ve seen them.

I’ve created a simple module to put in the URL for something I don’t want to forget. I’m then using a scraper to extract all information from the page to give me a little more information what the page is about.

#The future

I’m planning to add more features to the system. I’m thinking a daily digest which uses multiple sources like my iCloud reminders, calendar events, the weather, and the news to give me a daily overview of what’s happening and what I have to do. And also which goals I’ve missed yesterday and motivate me to achieve them today.

Turn websites into LLM-ready data
Power your AI apps with clean data crawled from any website. It’s also open-source.

Source

It’s a project that enables you to scrape and crawl websites and get well-formatted data. It also handles JavaScript rendering, which is a common issue when scraping websites. And the best part: it’s open source and you can host it yourself.

#Installing Firecrawl with Docker

Firecrawl provides everything you need to get started quickly. You may use the docker-compose.yaml file provided in their GitHub repository.

Make sure to clone the whole repository as the docker-compose.yaml file references other files in the repository.

Copy the .env.example file from the directory apps/api into the same directory where you have the docker-compose.yaml file and rename it to .env. If you don’t have reasons to change anything the default values work fine.

#Integrating with Traefik

If, like me, you are using Traefik as a reverse proxy, use the following example configuration to configure Traefik with labels.

This assumes you are using an external network called web to route all your traffic through Traefik.

I’ve used the headerauthentication plugin by omar-shrbajy-arive. This plugin allows you to define a specific header key and value that must be present in the request to access Firecrawl. You can also use it without authentication, but I prefer and recommend not to leave a web scraper open to the internet for anyone to use.

To use this plugin, add the following configuration to your Traefik static configuration file (traefik.toml).

1[experimental.plugins.headerauthentication]
2  moduleName = "github.com/omar-shrbajy-arive/headerauthentication"
3  version = "v1.0.3"

Now you can use the following docker-compose.yaml file to set up Firecrawl with Traefik, it is the exact same as the one provided by Firecrawl, but with the Traefik labels added.

  1name: firecrawl
  2 
  3x-common-service: &common-service
  4  build: apps/api
  5  networks:
  6    - backend
  7  extra_hosts:
  8    - "host.docker.internal:host-gateway"
  9 
 10services:
 11  playwright-service:
 12    build: apps/playwright-service
 13    environment:
 14      - PORT=3000
 15      - PROXY_SERVER=${PROXY_SERVER}
 16      - PROXY_USERNAME=${PROXY_USERNAME}
 17      - PROXY_PASSWORD=${PROXY_PASSWORD}
 18      - BLOCK_MEDIA=${BLOCK_MEDIA}
 19    networks:
 20      - backend
 21 
 22  api:
 23    <<: *common-service
 24    labels:
 25      - "traefik.http.middlewares.firecrawl.plugin.headerauthentication.header.name=Authorization"
 26      - "traefik.http.middlewares.firecrawl.plugin.headerauthentication.header.key=Bearer ${BEARER_TOKEN}"
 27      - "traefik.http.routers.firecrawl.rule=Host(`firecrawl.your-domain.com`)"
 28      - "traefik.http.routers.firecrawl.tls=true"
 29      - "traefik.http.routers.firecrawl.tls.certresolver=lets-encrypt"
 30      - "traefik.http.routers.firecrawl.middlewares=firecrawl"
 31    environment:
 32      REDIS_URL: ${REDIS_URL:-redis://redis:6379}
 33      REDIS_RATE_LIMIT_URL: ${REDIS_URL:-redis://redis:6379}
 34      PLAYWRIGHT_MICROSERVICE_URL: ${PLAYWRIGHT_MICROSERVICE_URL:-http://playwright-service:3000}
 35      USE_DB_AUTHENTICATION: ${USE_DB_AUTHENTICATION}
 36      PORT: ${PORT:-3002}
 37      NUM_WORKERS_PER_QUEUE: ${NUM_WORKERS_PER_QUEUE}
 38      OPENAI_API_KEY: ${OPENAI_API_KEY}
 39      OPENAI_BASE_URL: ${OPENAI_BASE_URL}
 40      MODEL_NAME: ${MODEL_NAME:-gpt-4o}
 41      SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL}
 42      LLAMAPARSE_API_KEY: ${LLAMAPARSE_API_KEY}
 43      LOGTAIL_KEY: ${LOGTAIL_KEY}
 44      BULL_AUTH_KEY: ${BULL_AUTH_KEY}
 45      TEST_API_KEY: ${TEST_API_KEY}
 46      POSTHOG_API_KEY: ${POSTHOG_API_KEY}
 47      POSTHOG_HOST: ${POSTHOG_HOST}
 48      SUPABASE_ANON_TOKEN: ${SUPABASE_ANON_TOKEN}
 49      SUPABASE_URL: ${SUPABASE_URL}
 50      SUPABASE_SERVICE_TOKEN: ${SUPABASE_SERVICE_TOKEN}
 51      SCRAPING_BEE_API_KEY: ${SCRAPING_BEE_API_KEY}
 52      HOST: ${HOST:-0.0.0.0}
 53      SELF_HOSTED_WEBHOOK_URL: ${SELF_HOSTED_WEBHOOK_URL}
 54      LOGGING_LEVEL: ${LOGGING_LEVEL}
 55      FLY_PROCESS_GROUP: app
 56    depends_on:
 57      - redis
 58      - playwright-service
 59    ports:
 60      - "3002:3002"
 61    command: [ "pnpm", "run", "start:production" ]
 62    networks:
 63      - web
 64 
 65  worker:
 66    <<: *common-service
 67    environment:
 68      REDIS_URL: ${REDIS_URL:-redis://redis:6379}
 69      REDIS_RATE_LIMIT_URL: ${REDIS_URL:-redis://redis:6379}
 70      PLAYWRIGHT_MICROSERVICE_URL: ${PLAYWRIGHT_MICROSERVICE_URL:-http://playwright-service:3000}
 71      USE_DB_AUTHENTICATION: ${USE_DB_AUTHENTICATION}
 72      PORT: ${PORT:-3002}
 73      NUM_WORKERS_PER_QUEUE: ${NUM_WORKERS_PER_QUEUE}
 74      OPENAI_API_KEY: ${OPENAI_API_KEY}
 75      OPENAI_BASE_URL: ${OPENAI_BASE_URL}
 76      MODEL_NAME: ${MODEL_NAME:-gpt-4o}
 77      SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL}
 78      LLAMAPARSE_API_KEY: ${LLAMAPARSE_API_KEY}
 79      LOGTAIL_KEY: ${LOGTAIL_KEY}
 80      BULL_AUTH_KEY: ${BULL_AUTH_KEY}
 81      TEST_API_KEY: ${TEST_API_KEY}
 82      POSTHOG_API_KEY: ${POSTHOG_API_KEY}
 83      POSTHOG_HOST: ${POSTHOG_HOST}
 84      SUPABASE_ANON_TOKEN: ${SUPABASE_ANON_TOKEN}
 85      SUPABASE_URL: ${SUPABASE_URL}
 86      SUPABASE_SERVICE_TOKEN: ${SUPABASE_SERVICE_TOKEN}
 87      SCRAPING_BEE_API_KEY: ${SCRAPING_BEE_API_KEY}
 88      HOST: ${HOST:-0.0.0.0}
 89      SELF_HOSTED_WEBHOOK_URL: ${SELF_HOSTED_WEBHOOK_URL}
 90      LOGGING_LEVEL: ${LOGGING_LEVEL}
 91      FLY_PROCESS_GROUP: worker
 92    depends_on:
 93      - redis
 94      - playwright-service
 95      - api
 96    command: [ "pnpm", "run", "workers" ]
 97 
 98  redis:
 99    image: redis:alpine
100    networks:
101      - backend
102    command: redis-server --bind 0.0.0.0
103 
104networks:
105  backend:
106    driver: bridge
107  web:
108    external: true

Please create an .env file which holds the value of the authorization bearer. For example:

1BEARER_TOKEN=your-token

#Conclusion

You should now have a fully functional Firecrawl instance running on your server. Make sure to point your domain to the server.

In a fast-paced work environment, it’s all too common to leave things undocumented, thinking we’ll remember the details later. But then life happens – maybe just the weekend, a vacation, or another project pulls us away. When we finally return to that task or project, we waste precious time reconstructing what was left unfinished. Documenting as you work isn’t just a nice-to-have – it’s essential for efficiency.

Documentation is like a breadcrumb trail you leave for your future self, and it’s also a safety net for your team. Whether you’re learning new skills, tracking task progress, or working on a project that might be passed to someone else, having organized notes can make all the difference. By blending professional and personal note-taking, you create a quick reference system that captures experience you don’t need to recall every day but might need another time. It’s also a well studied fact that writing things down helps you remember them better.

#Why Documentation is Essential for Efficiency

When we don’t document our work, we force ourselves to rely on memory – an inherently unreliable and time-consuming process. Here’s why documentation is so essential for maintaining an efficient workflow:

1. Continuity – When you step away from a project, even for a short break, your brain has to work to “reload” context when you return. Documenting lets you pick up right where you left off, eliminating the time it takes to remember what you were working on.
2. Team Collaboration – Documentation is crucial when you’re part of a team. It allows others to pick up where you left off, reduces the need for extra meetings, and prevents redundant efforts.
3. Skill Retention – Documentation doesn’t just apply to project tasks. When learning new skills or techniques, keeping a record ensures you have a quick reference to review instead of starting from scratch each time.

The goal is simple: you shouldn’t waste time on tasks you’ve already done. Document once, and you’ll save yourself (and your colleagues) time in the future.

#Organized, Purposeful Note-Taking

Documentation is most effective when it’s well-organized, purposeful, and accessible. That means using different systems for different types of information and maintaining a structured approach that makes it easy to know exactly where your information is stored. Here’s one way to approach it:

#Different Systems for Different Purposes

Use multiple documentation systems that align with the specific purpose of each type of note:

• Long-term documentation – Use a platform like Confluence or Google Drive for project documentation or information that needs to be accessible over time. This is where key project details, processes, and team-shared knowledge should live.
• Task-related documentation – Keep project and task-specific notes in a project management tool (like Asana, Trello, or Jira) so everyone involved has quick access to current task information. Make sure to always update these when new information is available.
• Personal notes – For skills you’re developing or quick, daily notes, use a tool that’s easily accessible to you, such as a note-taking app or physical notebook. This system should serve as a place for information that you’ll review, consolidate, or integrate over time.

#Make Daily Notes a Habit

Develop a habit of making daily notes to capture everything relevant that happens throughout your workday. If you prefer to use paper, consider digitizing these notes at the end of each day so they’re easier to search and reference later. These daily notes can act as a temporary “memory bank” until you review and integrate them into your long-term documentation system.

This also solves the following problems:

• You always have a place to jot down quick thoughts, ideas or meeting notes
• You can easily find information later when you need it
• You get into the habit of documenting as you go

#Regularly Review and Integrate Your Notes

To prevent information from getting stale or lost, review your notes periodically. During these reviews, integrate key points from your daily notes into your main documentation systems. For example:

• Summarize major insights or decisions from daily notes into project documentation.
• Add key learning points from personal notes into a more comprehensive skill-building document.

This process ensures your documentation is always up-to-date, clear, and relevant. It also reinforces your understanding and saves time later when you or others need to reference important information.

It’s also your touch point to create boilerplate templates for complex tasks or projects. This way you can start new projects faster and with less friction.

#The Goal: Know Where to Find Important Information

Your documentation system should make it easy to find what you need, exactly where you expect it. Storing task-related information in a personal notebook, for example, may prevent team members from accessing it when needed. Instead, having a reliable system where each type of note is stored appropriately allows you and your team to access crucial information without delay or confusion.

#Characteristics of Good Documentation

Good documentation isn’t just about capturing information; it’s about making it useful. Aim to create documentation that is:

• Clear – Use concise language that anyone on the team can understand.

• Concise – Avoid excess details; include only what’s essential for understanding the work.

• Transparent – Document the reasoning behind key decisions so that others understand not just what was done but why.

• Easily accessible – Store notes in places where they can be found when needed, using tags, folders, or categories as needed to keep everything organized.

• The specific meaning of these criteria will depend on the context – long-term documentation for a project, for instance, may require more detail and structure than a quick daily note. However, clarity and accessibility should be the guiding principles across all documentation efforts.

#Conclusion

Documenting as you go may seem like an extra step, but it’s a powerful investment in your future efficiency. By organizing your notes with purpose, maintaining consistency in your systems, and combining professional and personal insights, you create a reliable, searchable knowledge base. The time you save by not having to “relearn” or “rediscover” is time you can reinvest in moving forward, not retracing old steps.

As with the other skills: You need to develop a mindset for documenting and see it as a part of your workflow, not an extra task. The more you practice, the more natural it will become, and the more time you’ll save in the long run.

Imagine your keyboard is not just a tool but a toolbox. Typing speed and shortcuts are the basic tools you’ve learned to wield expertly, but sometimes you need more advanced instruments to get the job done efficiently. In this post, we’ll look at how certain productivity tools, such as Raycast (for Mac users) and other automation tools, can extend your tool belt and help you optimize your workflow.

Disclaimer: While Raycast is a Mac-specific tool, the principles we’ll discuss apply broadly. For Windows users, there are alternatives such as PowerToys and Wox, though functionality and integrations may vary.

#The Role of Workflow-Enhancing Tools

Shortcuts are excellent for speeding up simple, repetitive actions, but as tasks become more complex, the time savings from shortcuts alone become limited. Workflow-enhancing tools provide a solution by consolidating multiple steps into a single action, automating routines, or adding functions that aren’t otherwise accessible through native system shortcuts.

#Why Workflow Tools Matter

Tools like Raycast create an environment where:

• Automation reduces the need to manually repeat tasks.
• Centralization enables quick access to files, apps, and commands without leaving your workspace.
• Customization lets you shape your workflow around your unique needs, not the other way around.

These tools allow you to work faster, minimize distractions, and stay focused on the task at hand – all while keeping you in a “keyboard-first” mindset.

#Adding Functionality to Your Tool Belt

1. Universal Search and Access – Imagine being able to instantly find any file, document, app, or setting from one command center. Tools like Raycast allow you to search across your system without interrupting your current task.
2. Automating Repetitive Tasks – Many of us perform the same set of actions multiple times a day. A tool that allows you to automate these actions – like opening a set of apps for a particular project or executing specific commands – can save time and mental energy.
3. Integrating Third-Party Tools – Project management apps, version control, task tracking: Workflow tools allow you to interact with various apps directly, without needing to switch contexts.

#Building an Automation Mindset

One of the biggest benefits of these workflow tools isn’t just the time they save directly but the way they encourage you to think about automation. Here are a few ways to start using workflow tools with an automation mindset.

#Consolidate Tools with Integrations

Using tools like Raycast, you can interact with project management software, document storage, or even communication apps directly from the command center. This means you don’t have to switch between apps constantly, keeping your focus and minimizing digital clutter.

Tip: Try setting up a command or automation to retrieve project tasks or deadlines from a tool like Notion, Asana, or Trello. This keeps your workflow focused and reduces the need to open multiple tabs.

#For the Devs: Experiment with Small Custom Scripts

If you’re comfortable in a code editor, try writing simple scripts or commands to automate repetitive actions, like renaming files in bulk, running a quick search across documents, or organizing new files. Even basic scripting can bring a huge efficiency boost. Or go beyond and create your own custom Raycast extensions to solve your unique problems.

#Keeping the Keyboard as Your Anchor

One reason why tools like Raycast work so well for efficiency enthusiasts is that they keep the keyboard as the primary interface. This is the perfect evolution from mastering shortcuts: your hands stay on the keyboard, your attention stays on the task, and your workflow remains uninterrupted.

#Ideas for Keyboard-Centric Workflow Enhancements

1. Quick Calculations and Conversions – Instead of reaching for a calculator, a command center tool can handle basic calculations or unit conversions directly.
2. Clipboard History – Workflow tools often include clipboard managers, allowing you to access previously copied items without switching apps.
3. Access System Settings – Quickly adjust settings like brightness, volume, or Wi-Fi without leaving your keyboard.

This keyboard-centric approach helps maintain flow, limits distractions, and optimizes for speed.

Expanding your tool belt beyond shortcuts with workflow-enhancing tools brings your efficiency to a new level. By centralizing commands, automating routines, and enabling integrations, tools like Raycast provide capabilities that make complex workflows manageable and keep you focused on the bigger picture.

If you’re a Mac user, give Raycast a try and explore the possibilities it unlocks. For Windows users, tools like PowerToys, Wox, and Power Automate offer similar ways to streamline your tasks.

Efficiency doesn’t just stop at shortcuts – it grows with every tool you add to your belt. Start small, build the habit, and discover how much smoother and more productive your day can become.

Imagine a carpenter who needs 10 minutes just to hold his hammer correctly – unimaginable, right? Yet many office workers spend valuable time each day inefficiently using their basic tools like the keyboard and mouse. With just a bit of effort, you could work much faster and with less frustration. This first post is all about the basics for an efficient workflow and why I think it should be your top priority.

#The Keyboard – Your Most Important Tool

The keyboard is the centerpiece of your daily work. Every word, line of code, and almost every interaction with your computer relies on it. If you’re still “hunting and pecking” for each letter as you type, you’re leaving tremendous potential on the table. There’s hardly any other skill in an office environment that can make such a big difference as being able to type quickly and accurately.

#Work on Your Typing Speed

Whether you are a writer, a programmer, or an office worker, typing is a fundamental skill. The faster you can type, the more you can get done in a day. The average typing speed is around 40 words per minute, but with a bit of practice, you can easily double that. Just imagine how much time you could save if you could type twice as fast! Or imagine how much time you’ve wasted by not improving this skill earlier.

How to Learn to Type Faster

There are plenty of free tools to help you get faster and more accurate. I really liked Keybr for their simple and intuitive interface with a lot of information about your progress.

A few minutes of practice a day is all it takes to develop this skill over the course of a few weeks.

The hardest part is acknowledging that you need to improve. It’s easy to get stuck in your ways and think that you’re “good enough”. Just type a few words into Keybr and look at your speed. I’ll wait. The time you invest in improving your typing speed will pay off in the very near future. And since you’ve chosen to work in front of a computer: don’t you want to be good at it? Or do you want to be the carpenter who can’t handle his hammer?

Tip: Training doesn’t have to take long – 5 minutes a day is a great start. You’ll quickly see progress, and soon you’ll be typing without needing to look at the keyboard!

#Shortcuts – Small Helpers with a Big Impact

Clicking with the mouse may not seem like it takes much time, but over the course of a day, those seconds add up. If you perform the same action repeatedly – like copying and pasting text – it’s worth considering how much time a simple shortcut could save. A shortcut is like a secret “magic trick” that takes you straight to the action you need.

While you might and should be used to the most common shortcuts like Ctrl + C and Ctrl + V, there are many more shortcuts that can save you time and make your work more efficient.
Take the time to learn the shortcuts for your most-used applications – it’s a small investment that pays off quickly. Just take a look in the menu of your application – most of the time, the shortcuts are listed right next to the action or when you hover over the button with your mouse.

Tip: Don’t overwhelm yourself. Pick one or two new shortcuts each week and practice using them consistently until they become second nature.

Once you get used to shortcuts, your workflow will feel more natural and intuitive – less time spent clicking, more focus on what really matters. And you’ll look back at your old self, wondering how you ever got anything done without them.

#Mouse vs. Keyboard

The mouse is certainly useful, but many tasks are quicker to complete with the keyboard. This has two advantages: You save time and avoid moving your hand constantly between the mouse and keyboard – also a lot of little time savers. And it also helps to prevent repetitive strain injuries, which can be caused by using the mouse too much.

Take a closer look at how you’re using your keyboard and mouse this week – where are you losing time? Could a shortcut replace an action you do repeatedly with the mouse?

#Conclusion

Mastering your basic tools is like a carpenter mastering the hammer. The time you invest in developing these foundational skills will pay off in a smoother, more productive workday. Just a few minutes a day practicing the 10-finger method or learning new shortcuts can help you work faster and with less frustration. You’ll also stand out amongst your peers as someone who knows how to use their tools efficiently.

I also want to make clear again, that this is not about speed for speed’s sake. You owe it to yourself and your work to use your tools efficiently. It’s about working smarter, not harder. Ask yourself: would you trust a carpenter who can’t handle his hammer?

I’ve had some hoops to jump through to get everything working, so I thought I’d share the steps I took to get it up and running.

This guide assumes you are using an external network called web to route all your traffic through Traefik.

#Step 1: Follow the official installation guide

The official installation guide can be found here. Follow the guide to set up the required environment variables and start the Docker containers.

#Adjust the docker-compose.override.yml file

 1version: '3.4'
 2 
 3services:
 4    api:
 5        labels:
 6            - 'traefik.enable=true'
 7            - 'traefik.http.routers.librechat.rule=Host(`your.domain.example.com`)'
 8            - 'traefik.http.routers.librechat.tls=true'
 9            - 'traefik.http.routers.librechat.tls.certresolver=lets-encrypt'
10            - 'traefik.http.services.librechat.loadbalancer.server.port=3080'
11        networks:
12            - web
13            - librechat_default
14        volumes:
15            - ./librechat.yaml:/app/librechat.yaml
16 
17networks:
18    web:
19        external: true
20    librechat_default:
21        external: true

#Adjust the .env file

To get the keys you may use the following script, place it in a file and run the file with node file.js:

 1const crypto = require('crypto')
 2 
 3// Generate a 32-byte key (64 characters in hex)
 4const key = crypto.randomBytes(32).toString('hex')
 5 
 6// Generate a 16-byte IV (32 characters in hex)
 7const iv = crypto.randomBytes(16).toString('hex')
 8 
 9// Generate a 32-byte key (64 characters in hex)
10const jwt = crypto.randomBytes(32).toString('hex')
11 
12// Generate a 32-byte key (64 characters in hex)
13const jwt2 = crypto.randomBytes(32).toString('hex')
14 
15console.log(`CREDS_KEY=${key}`)
16console.log(`CREDS_IV=${iv}`)
17console.log(`JWT_SECRET=${jwt}`)
18console.log(`JWT_REFRESH_SECRET=${jwt2}`)

After that you can adjust the .env file:

 1HOST=localhost
 2PORT=3080
 3 
 4DOMAIN_CLIENT=https://your.domain.example.com
 5DOMAIN_SERVER=https://your.domain.example.com
 6 
 7OPENAI_API_KEY=your-key
 8ASSISTANTS_API_KEY=your-key
 9APP_TITLE="a fancy title"
10 
11CREDS_KEY=f151369e25852102edfa394fd034df5ac492c2a0028acaa51260402916488c65
12CREDS_IV=13272260bd313fd2d032ddcd70b75769
13JWT_SECRET=4d10e5b41de2d88a819b0e4b8600d1834c25356a44323cfb1bd3a8e839688b04
14JWT_REFRESH_SECRET=4f73e40aadf694f829b9980d0224cd14783d791f5744627af9d94ed71dc34943
15 
16ALLOW_REGISTRATION=true

#Starting the containers

After adjusting the files, you can start the containers with the following command:

1docker-compose up -d

That should be it! You should now be able to access Librechat AI at https://your.domain.example.com.

#Prerequisites

Before we start, you will need the following:

• A VPS with Ubuntu 22.04 with at least 1GB of RAM and 1 CPU core
• A domain name that you can point to your VPS
• A basic understanding of Linux and the command line

#Step 1: Install Typesense

After your server is spun up, SSH into it and run the following commands:

 1# Update the package list and the packages
 2apt update && apt upgrade -y
 3 
 4# Install Typesense
 5# x64
 6curl -O https://dl.typesense.org/releases/26.0/typesense-server-26.0-linux-amd64.tar.gz
 7tar -xzf typesense-server-26.0-linux-amd64.tar.gz
 8 
 9# arm64
10curl -O https://dl.typesense.org/releases/26.0/typesense-server-26.0-linux-arm64.tar.gz
11tar -xzf typesense-server-26.0-linux-arm64.tar.gz

#Step 2: Set up SSL

Before proceeding, make sure that your domain is pointing to your VPS.

To set up SSL, we will use Certbot to generate SSL certificates for our domain. Run the following commands:

1apt install snapd
2snap install core && snap refresh core
3snap install --classic certbot
4certbot certonly --standalone -d your.domain.example.com

This will also set up auto renewal of your certificates. To make sure that Typesense will use the new certificates you will need to restart the service when the certificates have been renewed.

With your preferred text editor, open the Certbot renewal configuration file /etc/letsencrypt/renewal/your.domain.example.com.conf.

Add the following line to the file:

1renew_hook = systemctl reload typesense-server.service

#Step 3: Configure Typesense

Open the Typesense configuration file /etc/typesense/typesense-server.ini with your preferred text editor and change the lines with the comments. All other lines can be left as they are.

1[server]
2 
3api-address = 0.0.0.0
4api-port = 443 # Adjust the port
5api-key = api-key # Set a secure API Key
6log-dir = /var/log/typesense
7ssl-certificate = /etc/letsencrypt/live/your.domain.example.com/fullchain.pem # Path to the SSL certificate
8ssl-certificate-key = /etc/letsencrypt/live/your.domain.example.com/privkey.pem # Path to the SSL certificate key

#Step 4: Adjust firewall

As a security measure, you should adjust the firewall to only allow incoming traffic on the ports you need. Run the following commands:

1ufw allow 443
2ufw allow 80
3ufw allow 22
4ufw enable

This ensures, that only traffic from these ports are allowed.

#Step 5: Start Typesense

You are now ready to start Typesense!

Run the command below to start Typesense:

1systemctl start typesense-server.service

You may check if everything worked by opening the health check URL in your browser: https://your.domain.example.com/health.

If everything worked, you should see a JSON response like this:

1{
2    "ok": true
3}

Since I’ve wanted to use OpenAI to generate text and already have a web service for this in place which is connected to a few other accounts I wanted to be able to generate the response in this service instead of having to write a plugin for Mattermost.

But for this to work I needed to be able to know when someone messaged my bot account.

Unfortunately the Outgoing webhooks integration only works in public channels, not in direct messages.

So I decided to write a little plugin for Mattermost which sends a webhook to my service when someone messages the bot account.

#The plugin

I’ve open sourced the plugin on GitHub. You can use this if you don’t want to write your own plugin.

For those interested I’ll explain the things which weren’t clear to me when creating the plugin because there was no clear documentation on this (or I couldn’t find it).

The plugin is based on the mattermost-plugin-starter-template, so I won’t explain the basics of how to create a plugin but only the things which are specific to this plugin.

Everything important happens in the file server/plugin.go.

#Adding configuration

I wanted to be able to quickly change the webhook URL without having to recompile the plugin. For this I added a configuration option to the plugin.

You do this by adding a struct called Configuration to the plugin struct:

1type Configuration struct {
2    BotUserID  string
3    WebhookURL string
4}

You can then add a new type for your plugin which embeds the Mattermost plugin and adds the configuration:

1type BotWebhookPlugin struct {
2    plugin.MattermostPlugin
3    configuration *Configuration
4}

To react to messages we’ll implement the MessageHasBeenPosted function. For easier reading I’ve removed the error handling from the code snippets.

 1func (p *BotWebhookPlugin) MessageHasBeenPosted(post *model.Post) {
 2  // get the channel from the post
 3  // we need this to check if the message was sent to the bot
 4    channel, err := p.API.GetChannel(post.ChannelId)
 5 
 6  // if the post was by the bot ignore it
 7    if post.UserId == p.configuration.BotUserID {
 8        return
 9    }
10 
11  // check if the message was sent to the bot
12  // the channel name in a direct channel looks like this:
13  // <bot username>__<user id>
14    if strings.Contains(channel.Name, p.configuration.BotUserID) {
15    // convert the post to JSON
16        jsonPayload, err := json.Marshal(post)
17 
18    // send a POST request to the webhook URL with the post as the body
19        req, err := http.NewRequest("POST", p.configuration.WebhookURL, bytes.NewBuffer(jsonPayload))
20        req.Header.Set("Content-Type", "application/json")
21        client := &http.Client{}
22        resp, err := client.Do(req)
23 
24        defer resp.Body.Close()
25 
26        return
27    }
28}

That’s all there is to it. Every time someone messages the bot the plugin will send a webhook to the configured URL.

In this article, I’ll walk you through my journey of using OpenAI’s GPT models, their embeddings, and a vector database called Weaviate to enhance our documentation search capabilities.

#Integrate OpenAI’s API with Laravel

We’ll assume you already have a Laravel application set up, or you’re familiar with setting up a new Laravel project. In this section, we’ll focus on integrating the OpenAI API using the openai-php/laravel Composer package.

#Step 1: Install the openai-php/laravel Package

To install the openai-php/laravel package, use the Composer command below:

1composer require openai-php/laravel
2php artisan vendor:publish --provider="OpenAI\Laravel\ServiceProvider"

#Step 2: Configure Environment Variables

Navigate to the root of your Laravel project and locate the .env file. This file contains environment-specific settings. We’ll need to add our OpenAI API key to this file. You can obtain an API key by signing up for an OpenAI account.

Add the following line to your .env file:

1OPENAI_API_KEY=sk-...

You are now ready to use the OpenAI facade in your Laravel application.

1use OpenAI\Laravel\Facades\OpenAI;
2 
3$result = OpenAI::completions()->create([
4    'model' => 'text-davinci-003',
5    'prompt' => 'I want to ',
6]);
7 
8echo $result['choices'][0]['text'];

#Retrieving Documentation Data from Confluence API

In this section, we will go through the process of fetching data from the Confluence API, which stores your documentation. We will be using this data for our natural language search.

#Step 1: Set Up Confluence API Credentials

To interact with the Confluence API, you will need an API token and your Confluence URL. You can create an API token by following the instructions in the official documentation.

Once you have your API token, add the following lines to your .env file:

1CONFLUENCE_API_USER=your_api_user_here
2CONFLUENCE_API_KEY=your_api_key_here
3CONFLUENCE_URL=https://your_domain.atlassian.net

Replace the values with your own values and make sure, the user you are using has access to the documentation you want to search.

Now add these values to your config/services.php file:

1'confluence' => [
2    'api_user' => env('CONFLUENCE_API_USER'),
3    'api_key' => env('CONFLUENCE_API_KEY'),
4    'api_url' => env('CONFLUENCE_API_URL'),
5],

#Step 2: Create a ConfluenceService Class

Create a new service class called ConfluenceService in the app/Services directory to handle interactions with the Confluence API.

Below is a simple service class which handles the fetching of all pages from a specific parent page in Confluence.

 1<?php
 2 
 3namespace App\Services;
 4 
 5use Illuminate\Http\Client\Response;
 6use Illuminate\Support\Facades\Http;
 7 
 8class ConfluenceService
 9{
10    private string $apiUser;
11    private string $apiKey;
12    private string $apiUrl;
13 
14    public function __construct()
15    {
16        $this->apiUser = config('services.confluence.api_user');
17        $this->apiKey = config('services.confluence.api_key');
18        $this->apiUrl = config('services.confluence.api_url');
19    }
20 
21    public function getPageDescendants(int $pageId): Response
22    {
23        $entityUrl = sprintf('content/%s/descendant/page', $pageId);
24        $data = [
25            'expand' => 'body.view',
26            'limit' => 1000,
27        ];
28 
29        $url = sprintf('%s/%s', $this->apiUrl, trim($entityUrl, '/'));
30 
31        return Http::withBasicAuth($this->apiUser, $this->apiKey)->get($url, $data);
32    }
33}

You may now use this service class to fetch data from the Confluence API.

1$confluenceService = new ConfluenceService();
2$response = $confluenceService->getPageDescendants(123456);

This response contains all subpages including their contents.

#Store embeddings of documentation in Weaviate

Weaviate is a vector database that allows you to store and query data in a vector space. You can read my article on how to set up Weaviate with Docker and Traefik to learn more about Weaviate and how to set it up. Read the article. If you have no previous experience or knowledge of Weaviate, I recommend reading the article first.

Weaviate uses a schema to define its data structure. In this example, we will use the following schema:

 1{
 2    "class": "Chunk",
 3    "description": "Some chunk of knowledge",
 4    "vectorizer": "text2vec-openai",
 5    "moduleConfig": {
 6        "text2vec-openai": {
 7            "model": "ada",
 8            "modelVersion": "002",
 9            "type": "text"
10        }
11    },
12    "properties": [
13        {
14            "name": "identifier",
15            "description": "The identifier of the particular chunk of knowledge",
16            "dataType": [
17                "string"
18            ],
19            "moduleConfig": {
20                "text2vec-openai": {
21                    "skip": true
22                }
23            }
24        },
25        {
26            "name": "content",
27            "description": "The contents",
28            "dataType": [
29                "text"
30            ]
31        },
32        {
33            "name": "source",
34            "description": "The source type",
35            "dataType": [
36                "string"
37            ],
38            "moduleConfig": {
39                "text2vec-openai": {
40                    "skip": true
41                }
42            }
43        },
44        {
45            "name": "sourceLink",
46            "description": "URL to the article",
47            "dataType": [
48                "string"
49            ],
50            "moduleConfig": {
51                "text2vec-openai": {
52                    "skip": true
53                }
54            }
55        }
56    ]
57}

We aren’t storing whole pages content because this would mean that our prompts will get too big. Instead we will chunk the content and store each chunk in Weaviate. We will also store the source and sourceLink properties to be able to link back to the original source.

#Step 1: Accessing Weaviate from PHP

To access Weaviate from PHP, we will use the weaviate-php package. Install the package using the Composer command below:

1composer require timkley/weaviate-php

You can now use the Weaviate client in your PHP code.

1<?php
2 
3use Weaviate\Weaviate;
4 
5$weaviate = new Weaviate('http://localhost:8080', 'your-token');

#Step 2: Chunk and store content in Weaviate

Looping over all our pages we’ll do the following things:

1. Remove all HTML tags from the content
2. Split the content into chunks manageble chunks
3. Create a new Weaviate object for each chunk
4. Store the object in Weaviate

 1use App\Services\ConfluenceService;
 2 
 3$confluenceService = new ConfluenceService();
 4 
 5$response = $confluenceService->getPageDescendants(12345);
 6 
 7if ($response->successful()) {
 8    $pages = $response->json()['results'];
 9 
10    foreach ($pages as $page) {
11        $content = cleanUpContent($page['body']['view']['value']);
12        $chunks = chunkContent($content);
13 
14        // Delete all old chunks before creating new ones
15        $weaviate->batch()->delete('Chunk', [
16            'path' => ['identifier'],
17            'operator' => 'Equal',
18            'valueString' => $chunkyBoy->identifier,
19        ]);
20 
21        $count = 0;
22        $objects = [];
23        // Loop over the chunks and create objects matching our Weaviate schema
24        foreach ($chunks as $chunk) {
25            $objects[] = [
26                'class' => 'Chunk',
27                'properties' => [
28                    'identifier' => $chunkyBoy->identifier,
29                    'content' => $chunk['value'],
30                    'source' => $chunkyBoy->source,
31                    'sourceLink' => $chunkyBoy->sourceLink,
32                ],
33            ];
34 
35            if (++$count % $batchSize === 100) {
36                $weaviate->batch()->create($objects);
37                $objects = [];
38                $count = 0;
39            }
40        }
41 
42        $weaviate->batch()->create($objects);
43    }
44} else {
45    // Handle the error
46    echo "Failed to fetch descendant pages: " . $response->status();
47}
48 
49function cleanUpContent(string $content): string
50{
51    return Str::of($content)
52        ->replace('<', ' <')
53        ->stripTags()
54        ->replace(['\r', '\n'], ' ')
55        ->replaceMatches('/\s+/', ' ')
56        ->trim();
57}
58 
59function chunkContent(string $content): array
60{
61    $tokensPerCharacter = 0.4;
62    $tokenLimit = 150;
63    $chunkCharacterLimit = $tokenLimit / $tokensPerCharacter;
64 
65    // Split the input string into an array of sentences
66    $sentences = collect(preg_split('/(?<=[.?!])\s?(?=[a-z])/i', $content));
67 
68    $chunks = $sentences->chunkWhile(
69        function (string $sentence, int $key, Collection $chunk) use ($chunkCharacterLimit) {
70            return $chunk->sum(fn (string $sentence) => strlen($sentence)) < $chunkCharacterLimit;
71        }
72    )->map(function (Collection $chunk) {
73        $value = $chunk->implode(' ');
74        $checksum = md5($value);
75 
76        return [
77            'checksum' => $checksum,
78            'value' => $value,
79        ];
80    });
81 
82    return $chunks->all();
83}

#Implementing the Natural Language Search

In this section, we’ll implement the natural language search feature using OpenAI’s GPT models and the Weaviate vector database. Our goal is to allow users to search the documentation using natural language queries, and return the most relevant results. Here’s how we’ll do it:

Assume we’ll have an endpoint that accepts a question parameter, you could implement this in your own application using the code below:

 1<?php
 2 
 3namespace App\Http\Controllers;
 4 
 5use Illuminate\Http\Request;
 6use OpenAI\Laravel\Facades\OpenAI;
 7use Weaviate\Weaviate;
 8 
 9class DocSearchController extends Controller
10{
11    public function search(Request $request)
12    {
13        $question = $request->input('question');
14 
15        if ($question) {
16            $chunks = $this->getChunks($question);
17            $messages = $this->getMessages($question, $chunks);
18 
19            $response = OpenAI::chat()->create([
20                'model' => 'gpt-3.5-turbo',
21                'messages' => $messages,
22            ]);
23 
24            $answer = $response['choices'][0]['message']['content'];
25        }
26 
27        return view('docsearch', [
28            'answer' => $answer ?? '',
29        ]);
30    }
31 
32    protected function getChunks(string $text): array
33    {
34        $weaviate = app(Weaviate::class);
35 
36        $query = <<<GQL
37        {
38          Get {
39            Chunk(
40              nearText: {
41                  concepts: "$text"
42                  certainty: 0.9
43              }
44              limit: 3
45            ) {
46                content
47            }
48          }
49        }
50GQL;
51 
52        $response = $weaviate->graphql()->get($query);
53 
54        if (isset($response['errors'])) {
55            return [];
56        }
57 
58        return $response ? $response['data']['Get']['Chunk'] : [];
59    }
60 
61    protected function getMessages(string $question, array $chunks): array
62    {
63        $information = implode('\n', array_column($chunks, 'content'));
64 
65        $messages = [
66            ['role' => 'system', 'content' => 'You are a helpful assistant.'],
67            ['role' => 'user', 'content' => 'Here is some information: ' . $information],
68            ['role' => 'user', 'content' => 'Please use this information to answer my question: ' . $question],
69        ];
70 
71        return $messages;
72    }
73}

#Conclusion

In this tutorial, I’ve demonstrated how to integrate OpenAI’s GPT models, Weaviate vector search, and Laravel to create a natural language search for your documentation. While this implementation works well, there are further optimizations we can apply to enhance the system and reduce API costs.

One area of improvement is only updating the embeddings of your documentation content when needed. Embeddings can be expensive to compute, both in terms of time and API costs. By only updating them when the content changes, you can save on API bills and improve response times.

To achieve this, you could use MD5 hashes to check whether the content has changed or not. When you receive a new content update, calculate its MD5 hash and compare it to the hash of the previous content. If the hashes are different, update the embeddings in Weaviate and store the new hash for future comparisons. This way, you’ll only update the embeddings when there’s an actual change in the content.

By applying these improvements, you’ll create a more efficient and cost-effective natural language search system for your documentation, while maintaining a high level of accuracy and relevance for your users.

Although there’s no apparent method to do this in the UI, it’s surprisingly easy to accomplish!

#Steps to reconnect an unavailable Zigbee device

1. Open the Settings → Devices & Services page
2. Click on “Add integration”
3. Select “Add Zigbee device”
4. Put your unavailable device into pairing mode

Your device should now re-pair and be available again.

The advantage of this method is that you don’t have to remove the device from Home Assistant first, which would break all existing automations.

Update 2023-04-15: Since version 1.18 of Weaviate you can use API key authentication, thus making the header authentication plugin unnecessary.

#What is Weaviate?

Weaviate is an open-source vector search engine. It allows you to store data objects and vector embeddings from your favorite ML-models, and scale seamlessly into billions of data objects.

Source

In other words, it’s a search engine for machine learning models. It’s a helpful tool for building your own AI applications. You can use it to store and search for data objects using vector embeddings. It also integrates directly with OpenAI, allowing you to use their embedding models directly upon importing text.
This enables you to do very easy near-text-searches because the query is also automatically embedded.

#Installing Weaviate with Docker

To install Weaviate with Docker, you can use the official Docker Compose Configurator. This will provide you with a docker-compose.yml file that if perfectly pre-configured to your liking.

Just remember to set a persistent volume if you don’t want to lose your data when you restart the container.

#Integrating with Traefik

If, like me, you are using Traefik as a reverse proxy, you can use the following docker-compose.yml file to set up Weaviate with Traefik.

This assumes you are using an external network called web to route all your traffic through Traefik. The rest of the file was created using the above-mentioned configurator. Depending on your preferences the environment section may vary.

I’ve used labels to configure Traefik for this container.

 1---
 2version: '3.4'
 3services:
 4  weaviate:
 5    image: cr.weaviate.io/semitechnologies/weaviate:1.17.4
 6    ports:
 7    - 8080:8080
 8    restart: on-failure:0
 9    volumes:
10      - /var/weaviate:/var/lib/weaviate
11    labels:
12      - "traefik.http.routers.weaviate.rule=Host(`weaviate.your-host.com`)"
13      - "traefik.http.routers.weaviate.tls=true"
14      - "traefik.http.routers.weaviate.tls.certresolver=lets-encrypt"
15    environment:
16      OPENAI_APIKEY: $OPENAI_APIKEY
17      QUERY_DEFAULTS_LIMIT: 25
18      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
19      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
20      DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
21      ENABLE_MODULES: 'text2vec-openai,generative-openai'
22      CLUSTER_HOSTNAME: 'node1'
23    networks:
24      - web
25 
26networks:
27  web:
28    external: true

#Adding a simple header authentication

I didn’t want to expose Weaviate to the internet without some kind of authentication. While Weaviate supports OIDC authentication, this seemed overkill for my use case. I just wanted to add a simple header authentication to prevent unauthorized access.

To achieve this, I’ve used the plugin headerauthentication by omar-shrbajy-arive. This plugin allows you to define a specific header key and value that must be present in the request to access Weaviate.

To use this plugin, you need to add the following to your Traefik static configuration file, assuming you are using the toml format:

1[experimental.plugins.headerauthentication]
2  moduleName = "github.com/omar-shrbajy-arive/headerauthentication"
3  version = "v1.0.3"

Then you can add the following to your docker-compose.yml file:

 1---
 2version: '3.4'
 3services:
 4  weaviate:
 5    image: cr.weaviate.io/semitechnologies/weaviate:1.17.4
 6    ports:
 7      - 8080:8080
 8    restart: on-failure:0
 9    volumes:
10      - /var/weaviate:/var/lib/weaviate
11    labels:
12      - "traefik.http.routers.weaviate.rule=Host(`weaviate.wacg.dev`)"
13      - "traefik.http.routers.weaviate.tls=true"
14      - "traefik.http.routers.weaviate.tls.certresolver=lets-encrypt"
15      - "traefik.http.routers.weaviate.middlewares=weaviate@docker" 
16      - "traefik.http.middlewares.weaviate.plugin.headerauthentication.Header.name=X-TOKEN" 
17      - "traefik.http.middlewares.weaviate.plugin.headerauthentication.Header.key=${HEADER_TOKEN}" 
18    environment:
19      OPENAI_APIKEY: $OPENAI_APIKEY
20      QUERY_DEFAULTS_LIMIT: 25
21      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
22      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
23      DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
24      ENABLE_MODULES: 'text2vec-openai,generative-openai'
25      CLUSTER_HOSTNAME: 'node1'
26    networks:
27      - web
28 
29networks:
30  web:
31    external: true

This configures a new middleware called weaviate using the headerauthentication plugin. It also adds a new environment variable called HEADER_TOKEN that will be used to set the value of the header key.

Please create an .env file which holds the value of the header key. For example:

1HEADER_TOKEN=your-token

#Conclusion

You now have a fully functional Weaviate instance running on your server. You can use it to build your own AI applications.

If you’ve been using Docker for a while, you may have noticed that it can quickly use up a lot of disk space. Especially after testing out a few images, updating existing ones you may find your host slowly but surely filling up.

In this quick tip, I’ll show you how to use the built-in command docker system prune to clear up space used by Docker.

#What does docker system prune do?

To quote the official documentation: this command “removes unused data”.

In practice, this means that it will remove:

• all stopped containers
• all networks not used by at least one container
• all dangling images
• all build cache

I’ve found, that especially dangling images can use up a lot of space. If you are using my quicktip for updating your Home Assistant installation this can quickly use up a lot of space on your SD card.

#Available options

--all, -a - Remove all unused images not just dangling ones

--force, -f - Do not prompt for confirmation

--volumes - Prune volumes

#How to use it

To use the command, simply run it in your terminal with your wanted options. For example, to remove all unused images and dangling containers, you would run:

1docker system prune --all

#Conclusion

Using the built-in command docker system prune is a quick and easy way to clear up space used by Docker. It’s a good idea to run it every once in a while to keep your host running smoothly and efficiently. On my Raspberry Pi I run it after updating my existing containers.

Pipedream is a powerful and flexible platform for building and automating workflows. It allows you to connect a wide variety of apps and services, and trigger actions based on events or data from those services. With Pipedream, you can automate all kinds of tasks, from sending emails and texts to updating social media or setting reminders.

One example of how you can use Pipedream is to automatically set Mattermost’s DND (Do Not Disturb) mode when a Google Calendar event starts. This can be especially useful if you have meetings or other events that you don’t want to be interrupted during. By using Pipedream to automate this process, you can save time and hassle, and ensure that you remain focused and productive during your events.

In this blog post, I’ll walk you through the process of setting up this workflow in Pipedream. I’ll show you how to set up a Google Calendar trigger, how to set up a Mattermost action, and how to put it all together to create a complete workflow. By the end of this post, you’ll be able to automate the process of setting Mattermost’s DND mode when a Google Calendar event starts, using Pipedream.

#Setting up the accounts

To get started with setting up a Pipedream workflow to automatically set Mattermost’s DND mode when a Google Calendar event starts, you’ll need to have a few things in place:

1. A Pipedream account: If you don’t already have a Pipedream account, you can sign up for one at pipedream.com. It’s free to get started, and you can choose a paid plan later if you need more compute time. But Pipedream’s free tier is more than enough to get started with this workflow.

2. A Mattermost instance with the right to create a Personal Access Token, which is used to authenticate against the REST API. Your instance administrator can help you with this if you don’t have the right to create a Personal Access Token.

3. A Google Calendar account: Finally, you’ll need a Google Calendar account to use as the trigger for your workflow.

Once you have these accounts set up, you can follow these steps to connect them to Pipedream:

In Pipedream, click on the “Connections” tab in the left-hand menu.

1. Click on the “Accounts” link, click on the “Connect an app” button, and choose “Google Calendar” from the list of available connections.

2. Follow the prompts to connect your Google account to Pipedream. You’ll need to provide your Google login credentials and authorize the connection.

#Building the workflow

Click on “Workflows” in the sidebar menu and click on the “New +” button in the top right corner to create a new workflow.

#Setting up the Google Calendar trigger

After starting to create a new workflow, the first thing to do is add a trigger. In this case, we’ll use a Google Calendar trigger. This trigger will fire whenever a new event is added to a specific calendar.

Select the Google Calendar account you want to use as the trigger, and choose the specific calendar you want to use. You should now also add a timer. Pipedream will use this timer to check for new events every x minutes. I’ve used 25 minutes because I had problems with the default 15 minutes with very short events. You can adjust this to your needs.

Your trigger should now look like this:

Click on the “Create source” button to move on to the next step.

At this point, you should have a basic Google Calendar trigger set up in Pipedream. When an event starts on the calendar you selected, this trigger will be activated and the workflow will continue to the next step.

#Adding a Node action

The next step is crucial for this workflow to work perfectly. Unfortunately the event is not triggered in the exact moment the event starts. This is because Google Calendar doesn’t provide a trigger for this. Instead, you’ll get a trigger for an event that starts in the next 25 minutes (or whatever timer setting you used). This is not ideal, but it’s the best we can do with the current Google Calendar API.

To solve this, we’ll use a Node action which will parse the event start and end time to delay the workflow until the event starts. Here is the code we’ll use:

 1export default defineComponent({
 2    async run({ steps, $ }) {
 3        const eventStartDate = Date.parse(steps.trigger.event.start.dateTime)
 4        const eventEndDate = Date.parse(steps.trigger.event.end.dateTime)
 5 
 6        const triggerStartDate = Date.parse(steps.trigger.context.ts)
 7 
 8        const delay = eventStartDate - triggerStartDate
 9 
10        if (delay < 0) {
11            return $.flow.exit('The event already started')
12        }
13 
14        $.export('delay', delay)
15        $.export('dnd_end_time', eventEndDate)
16    },
17})

We’ll concentrate on the part inside the run function, the rest is normal boilerplate code. The first two lines parse the event start and end time to a JavaScript Date object.

The third line parses the trigger start time to a Date object.

With these variables in place we can now calculate how much we have to delay the workflow to continue at the exact moment the event starts.

Let’s subtract the trigger start time from the event start time. This will give us the delay in milliseconds. We’ll store this in the delay variable.

If the delay is under 0 we use an early return to exit the workflow in case the event already started.

We need to export the delay and the end time for the event. We’ll use these in the next step.

#Delaying the workflow

The next step is to delay the workflow until the event starts. To do this, we’ll use the Delay action. This action will delay the workflow for a specified amount of time. We’ll use the delay we calculated in the previous step.

Make sure the set the unit to milliseconds.

#Set the DND mode in Mattermost

After you’ve obtained your personal access token you are ready to build up the HTTP request we are going to make to the Mattermost REST API.

Use the official guide for Personal Access Tokens to obtain your own token. If you can’t create a token of your own, you might need to ask your Mattermost administrator to create a token for you or to give you the right to create one.

Add the action “HTTP Request” with the method PUT and the endpoint /api/v4/users/me/status with the following body:

1{
2    "user_id": "me",
3    "status": "dnd",
4    "dnd_end_time": "{{steps.node.dnd_end_time / 1000}}"
5}

Note that we set a key of dnd_end_time. This is used to automatically end the DND mode when the event ends.

Your body section should now look like this:

Make sure you use your personal access token in the auth section using the Bearer Token authorization type.

You’re good to go! After you’ve saved the workflow, it will be triggered whenever a new event is added to the calendar you selected. The workflow will delay itself until the event starts and then set the DND mode in Mattermost.

#Bonus: Set a custom status with the event name

If you want you can also set a custom status with the event name. To do this, you’ll need to add another HTTP request action. This time, you’ll need to set the method to PUT and the endpoint to /api/v4/users/me/status/custom. You’ll also need to set the body to the following:

1{
2    "emoji": "calendar",
3    "text": "{{steps.trigger.event.summary}}",
4    "expires_at": "{{steps.trigger.event.end.dateTime}}"
5}

Note: the API is a bit inconsistent here. The dnd_end_time is in seconds, but the expires_at is in milliseconds. This is why we needed to divide the dnd_end_time by 1000.

Since there currently is no direct migration path from MySQL to PostgreSQL which is officially vetted by Mattermost, to following path outlines how you can manually migrate a Mattermost instance by following the steps outlined here.

#Migration outline

1. Upgrade current in use Mattermost instance
2. Setup new server
3. Export data from old server
4. Import data to new server
5. Migrate settings and external integrations

#1. Upgrade current in use Mattermost instance

Your first step should be to make sure that your current instance is using the latest Mattermost version or at least version to which you are migrating. This is important because data exports may be incompatible if you try to import a newer export into an older server or the other way around.

You can find instructions on how to upgrade in the official Mattermost docs.

#2. Setup new server

I’ve used the Mattermost Omnibus installation method. If you’ve used another method the results of the following steps may vary. But since we will be using the command line utilities, which come packaged with Mattermost regardless of method of installation, the next steps should work the same.

#3. Export data from old server

Now you’ll use the mmctl command line tool to export the data from the old server with the following commands:

First, you’ll need to authenticate the command line tools with your server:

1mmctl auth login https://oldmattermosturl.com

At this point you should shut off access to the old server so no new data is generated. Do this on the weekend or after-work hours.

The following command will create an export job. The command returns the id of the created export which you can use to check the status of the export.

1mmctl export create

Use the export job show command to check the status of your export by replacing {id} with the id from the previous command.

1mmctl export job show {id}
2 
3--- Output
4ID: j9fgs4h7htpu0nl14fir3ncsno
5Status: success
6Created: 2022-10-28 10:55:25 +0000 UTC
7Started: 2022-10-28 10:55:31 +0000 UTC
8Data: map[include_attachments:true]

After the status changes to success, use the next command to show all available exports. This will show you all finished exports.

1mmctl export list
2 
3--- Output
4j9fgs4h7htpu0nl14fir3ncsno_export.zip
5 
6There are 1 exports on https://oldmattermosturl.com/

Copy the filename from the previous command to download the export. This will download the file into the current directory from which you’ve started this command.

1mmctl export download {filename}

You should now have the data export on your old Mattermost server. Depending on your infrastructure and if your old and new server have access to each other you will now need to somehow transfer the zip file to the new server.
In my case I used rsync to transfer the data to my laptop:

1rsync -av oldmattermosthost:/path/to/exported-file.zip .

#4. Import data to new server

Using rsync again, I transferred the zip file to the new host:

1rsync -av ./exported-file.zip newmattermosthost:/path-where-the-file-should-be

Again, the first thing you’ll need to do is authenticate the command line tool with on your new server - I’ll assume you already installed Mattermost on the new server.

1mmctl auth login https://newmattermosturl.com

Depending on how big your data export file is you will need to make a few adjustments on the new instance as well as the server. For our team of around 15 people and an instance of around 18 months old this amounted to a backup size of around 5GB - including attachments, which is much too big for the default settings.

You will have to do 2 things:

• bump the max file upload size in the system console of Mattermost
• update the nginx config to allow bigger upload sizes

Add an admin account and open the system console. You will find the maximum file size setting by either searching for it or navigate to Environment → File Storage → Maximum File Size. Set a maximum file size that is bigger than your backup.

If you don’t do this, you will get an error message like this:

Error: failed to create upload session: Unable to upload file. File is too large.

Next, edit the nginx config file for mattermost (located at /etc/nginx/conf.d/mattermost.conf) and configure the client_max_body_size to be bigger than your zip file. There are multiple locations where you have to overwrite this setting.
I used sed to replace all occurrences.

If this setting is too small you will get the following error message:

Error: failed to upload data: AppErrorFromJSON: model.utils.decode_json.app_error
413 Request Entity Too Large

1sed -i -e 's/50M/4G/' /etc/nginx/conf.d/mattermost.conf # replace 4G with what matches your file size
2systemctl reload nginx

The next step is to upload the zip file to Mattermost. Although this seems counterintuitive at first glance, it’s a necessary step to make the backup available for the import command itself.
Execute this command with the correct path and filename to your previously rsynced file.

1mmctl import upload exported-file.zip
2 
3--- Output
4Upload session successfully created, ID: 903p9ttsslrfpryorde2ah25bc
5Import file successfully uploaded, name: qjtchzygoalp5whzf4p4ah36nb

Depending on the size of the file this could take a while.

Next, run this command to see all available imports. This will return the filename for the following command.

1mmctl import list available
2 
3--- Output
4903p9ttsslrfpryorde2ah25bc_qjtchzygoalp5whzf4p4ah36nb_export.zip

Copy the filename of the file.
This is the part where your exported data will actually be restored into your new Mattermost instance:

1mmctl import process {filename_from_previous_command}

This will take a little while longer than the upload because all of your data is now restored. The job is running in the background but you can check on it with the following command:

1mmctl import job show {id}

After the job has finished all your data is restored. Because this only restores you have one last step left for a complete migration.

#5. Migrate settings and external integrations

There is one problem with this method: since it’s not a full database backup, it won’t restore your settings and integrations.
So you will have to manually migrate / setup these things to match your old instance. If you setup new integrations keep in mind that things like webhook urls and API tokens are regenerated, so you need to swap them out in your 3rd-party tools, also.

One last thing I’ve noticed: user images aren’t migrated, too. But this should be manageable :) It gives your users the chance to finally upload a current picture of themselves.

#Create a bot integration

As an admin, select the grid icon in the top left of your Mattermost instance and select “Integrations”.

You will have a couple of options to choose from:

You’ll want to select “Bot Accounts” and add a new bot account by clicking on “Add Bot Account”. Next, you will have to fill out a form. If you need more information about each of these fields, or bot accounts in general, I highly recommend the official documentation.

After saving the form you will receive the bot token. Copy and save this someplace safe as you will not see this again. You can always generate a new token but will have to replace the token everywhere you have used it.

Now to the fun part: sending messages with your shiny new bot! We will use Mattermost’s REST API to achieve this.

To access the API you will need the token you got in the previous step. Every request should send the following header, also known as Bearer or Token authentication: Authorization: Bearer your-token.
The base url for the API at the time of writing this article is:
https://your-mattermost-url.com/api/v4.

To send a message from a bot to a user you will now have to make two API requests: one that gives you the correct channel id and another one which actually sends the message.

#Getting the bot and the user id

Writing a message from a bot to a user basically means that you send a message to the private channel shared by these two entities. Every channel in Mattermost has its own unique id. And direct messages are nothing else as a private channel between two people.

If we want to get the channel id for the combination of the bot and the user we will need their own entity ids to query the API for the correct channel id.

Getting the user id is simple: you can just open the System Console, and click on “Users” in the submenu “User Management”. The user id is directly below the name of your users.

But a bot is no normal user! So the user list unfortunately doesn’t show the bot account. But you can use the bot token to use the endpoint GET /users to retrieve all users. You can then filter down the result to the username of your bot.
Alternatively, if you have a lot of users, use the endpoint POST /users/search.

Use this JSON payload to search for your bot:

1{
2    "term": "name-of-your-bot"
3}

Great! Now that you have the id of your bot and your user we can move on to the next step.

#Getting the direct channel id

Use the endpoint POST /channels/direct to get the correct channel id to send our message to.

Using the obtained bot and user id we can now send the following JSON payload to the named endpoint.

1[
2    "bot-id",
3    "user-id"
4]

Response

1{
2  "id": "this-is-the-channel-id",
3  "name": "first-user-id__second-user-id",
4  ...
5}

Note down this id, that is the unique identifier of the private channel between the bot and the specified user.

#Sending a message

Now we have all we need to finally send a message.

Lets now use the endpoint POST /posts to create a new message.

The minimal payload for this looks like this:

1{
2    "channel_id": "this-is-the-channel-id",
3    "message": "Very thoughtful message."
4}

If everything worked you will get a status 201 Created back.

#Real life example

At my work we are using such an integration to make users aware of problems with their project planning. There is a cronjob running every 15 minutes which checks the tasks of every user. If it detects new problems we can use the associated mail address to find the user, find the private channel of the bot and the user, and send them some meaningful message and a link to the planning tool to show what actual problems occurred.

I wanted a quick way to resize all these images to one output format, defined as the following:

• end image should be 500 x 500 pixels
• background should always be white
• images should have a slight border of 15 pixels
• images should be centered

For this task I used the free software ImageMagick, specifically its CLI tool mogrify

On Mac you can use brew to install the CLI with brew install imagemagick.

Our command will look like this:

1mogrify -path processed -trim -resize "470x470>" -gravity center
2-extent 500x500 -background white -format jpg "*"

Let’s break this command down bit by bit:

-path: Path where the image should be put out (“processed” in this case) -trim: Cut the image on all sides so not whitepsace is left (if the image sits on white or transparent background) -resize: width x height, > means the longest side will be used for resizing, the other side will be resized proportionally -gravity: Parameter order is important as this affects the next parameter (extent). Defines how extent works -extent: Extends the image to 500 x 500 pixels, our final format. Since gravity is set to center it gets extended equally on all 4 sides. -background: Define a background color -format: Defines the format "*": Double quotes ensure that the wildcard glob is not expanded by the shell, but mogrify itself.

If you now run this command in a folder with all your images you will have perfectly square images which can be directly used without any CSS tricks.

Maybe you also need a small API like this and want to get a better grasp how to do this. In this article we’ll use Netlify functions to write a simple, globally accessible API that returns some arbitrary value.

#First steps

I’ll assume you already have an account with Netlify and maybe already published something with it. If not, I recommend the “Add Site“ section of Netlify’s documentation.

Next, we’ll install the Netlify CLI. Although not a hard requirement, I highly recommend installing it, because this will allow you to test your functions locally.

Make sure you have Node installed and run the following command:

1npm install netlify-cli -g

This installs the Netlify CLI globally and makes the netlify command available to you.

Next, authenticate yourself using the command netlify login and then we are ready to write our first function!

#Simple Hello World! function

Create a new folder for this project and inside it put a file into the folder netlify/functions called hello-world.js. Netlify will look into this folder by default for functions to deploy. After deployment these function will be available at your-domain.com/.netlify/functions/name-of-your-file-without-extension.

Your folder structure should now look like this:

1hello-world
2└── netlify
3    └── functions
4        └── hello-world.js
5 
62 directories, 1 file

From the hello-world directory run the following command:
netlify functions:serve. This will locally serve your functions and the output should look like this:

1 ~/Code/hello-world ▶ netlify functions:serve
2◈ Loaded function hello-world.
3◈ Functions server is listening on 9999

The local server is now available on http://localhost:9999.

If you open the function in the browser you will see an error:

This is because our file is still empty and no function was exported. Every file must export an async handler function, like this:

1exports.handler = (event, context) => {
2    // ...
3}

The function accepts two parameters: event and context. The event parameter is an object which contains information about things like which http-method was used, the query-string, etc.
You can read more about the specific contents in the Netlify docs.

The function must return an object which is compatible with the JavaScript Response Object. So, your hello-world example could look like this:

1exports.handler = async (event, context) => {
2    return {
3        statusCode: 200,
4        body: 'Hello World!',
5    }
6}

If you now visit the URL http://localhost:9999/.netlify/functions/hello-world you should see Hello World! in your browser. If you pushed these changes to your repo now (and have already configured auto-deploy in Netlify) you could also access this publicly.

Now that you have a working function we can add some useful functionality.

#Random quote API

Instead of jokes let’s return some Star Wars jokes. Rename your hello-world.js file to star-wars.js and add the following content:

 1exports.handler = async (event, context) => {
 2    const quotes = [
 3        'I find your lack of faith disturbing.',
 4        'Do. Or do not. There is no try.',
 5        'A long time ago in a galaxy far, far away...',
 6    ]
 7 
 8    const randomQuote = quotes[Math.floor(Math.random() * quotes.length)]
 9 
10    const response = JSON.stringify({ quote: randomQuote })
11 
12    return {
13        statusCode: 200,
14        body: response,
15        headers: {
16            'Content-Type': 'application/json; charset=utf-8',
17            'Access-Control-Allow-Origin': '*',
18        },
19    }
20}

Let’s talk about what’s happening here:

First, we assign some quotes to an array and get a random quote by using Math.random().

Next, we’ll use JSON.stringify() to transform an actual object to a string which we can send with the response.