How to make a package manager (for a custom language!) — Part 2: The Command-Line Tool
This is part 2 of a series on how to build a package manager. If you haven't gone through part 1, check it out here.
Part 1 now contains all 3 parts, so go there.
Let's quickly go over what we've done so far.
- We've built a CDN, which stores package data and will deliver it to our tool.
- Our CDN responds to
GETrequests on the url
/package/<packagename>/<version>/<file>. Any invalid fields will result in a 404 error and a message such as "This package does not exist".
So, our task at hand: build a command-line tool which gets the package data from the CDN and places it in a nice location. For this tutorial we'll use
Let's get started! (The example repl can be found at https://repl.it/@fuzzyastrocat/cmd-line. As with before, feel free to follow along if you like.)
For this section, I'm going to use Go. I'm using Go because of two reasons:
- Go works nicely for HTTP things
- Go is compiled
This second one is especially important here — we need our tool to be compiled since we want to be able to create an executable from it. So, as with last time, it's okay if you want to use a different language, but consider carefully based on the two criteria above.
Implementation of the Command-Line Tool
First things first, we need to set up our repl.it workspace. Make a new Go project. Inside our new Go project, create a file called
.replit. This is a magical special file that repl.it looks at when you click the run button. In it, put the following:
This tells repl.it that our repl's main language is Go and that when we click the run button it should execute
go build -o packager -ldflags \"-s -w\" main.go. (The
ldflags just tell the go compiler not to generate version or debug info embedded in the executable. We don't really need to worry about that info since this will become a standalone executable.)
-o packager tells Go that we want to output our new executable to
./packager. DO NOT remove this or use
-o main: repl will hide an executable called
./main by default, so you won't realize that the program has compiled!
.replit file overrides repl's default behavior, which is to run the go file. Instead, we simply
build it, meaning that an executable (
./packager) is generated. We can then run this executable by typing
./packager in the shell (on the right). When you are done with your packager, this will be the actual tool — that way, you can run it without any need for the
Okay, now that we've set up our project let's start implementing our interface. The first thing we need to do is check what package the user wants. When you run the executable like
./packager install some_package,
some_package will get given to Go in the form of an array called
os.Args. To use it, we need to import
os. Let's make a quick program that just echoes the arguments we give it:
Now, click the Run button of your repl. Wait for it to finish running — you should see a file named "packager" get generated. Once it's done building, type
./packager these are some arguments into the shell on the left — the program should print those arguments back to you.
"Wait, why do we ignore the first argument?" Remember that the first argument will always be
./packager, and we don't care about the name of the program.
Okay, let's do something more useful with those arguments. We'll expect arguments of the following form:
We'll start by making the only option
install — we include this only for later extensibility. Let's simply start by printing out the available versions of a package:
Obviously, replace "your-cdn-url.repl.co" with your actual cdn url from the last tutorial.
Run the repl, and try
./packager install foo. It should respond with "That package does not exist!". Try again with
./packager install example (assuming that you followed the previous tutorial), and it should print out
Okay, now we need to figure out how to parse this array. We'll make use of Go's
encoding/json package, since it parses JSON arrays nicely:
versions is a
byte, which is what
json.Unmarshal takes as an argument, so we don't do any
string() conversion. If all goes correctly,
./packager install example should now print out "Versions: [1.0.0]".
Now we need to figure out how to find the latest version. We'll assume that all versions all well-formed sequences of 3 numbers separated by a
. (things like
2.10.0 are fine but
0.5 are not), which makes writing a compare function easier:
Now, we just need to iterate through the array of versions to find the latest (we assume there is at least one version):
So now we can tie it all together to find the files in the package:
Once again, replace
your-cdn-url.repl.co with your actual cdn url. If running
./packager install example prints out
["main.lang"] (or whatever your example filename was), great! We can now move on to actually (finally!) fetching the files.
First, we need to parse the array. Then, for each file in the array, we need to fetch (http GET) the contents of the file, which we can then save in a local directory. First though, we'll just print the contents of each file. Replacing
fmt.Println(string(files)), here is the relevant code to complete this task:
your-cdn-url.repl.co as needed.) I won't explain this in detail because there's really no new code here. We're just doing what we've done before, except extended slightly.
So... try it out! Last tutorial, I put
Test file! as the contents of
main.lang. Therefore, I should expect
Test file! to be printed to the console (which it was). If your program does this correctly, then we can move on to the final section: doing something useful with this file data.
First of all, we need to create a folder called
./pkg_data if it doesn't exist. This is easy with
(This code goes at the top of
main(), you'll see it in the completed code linked below.)
0755 is a permissions number, it means that you'll be able to do anything with the folder and other users can only read it and enter it (but not modify it).
After we get the
os.Args, we need to create a folder for the package. We can do that in the same way as above, with
Now we can finally write the files into our local package location. We can write to and make a file with
fmt.Println(string(contents)).) Okay, the big moment has come: try building and running
./packager install example. If all goes well, you should have a file
./moduledata/example/, with the contents you put in your CDN!
Completed code can be found here. Obviously, replace all occurrences of
pkg-cdn.fuzzyastrocat.repl.co with your CDN url.
Wow, that was a lot of work for an utterly trivial result! But now, we do have a minimal package manager which can fetch and "clone" packages from our CDN. Play around with the CDN — try adding more files to the
example package and use your command line tool to fetch them. Just remember to re-run your CDN (not the command line tool!) every time you add or change hosted package files. (Note that right now our
install option both installs and/or updates, we'll separate out that duty later.)
The nice thing about using a compiled language is that you can take the
./packager executable, put it anywhere (within the same type of system, obviously a Windows-generated executable won't work on a Mac), rename it, do whatever (without modifying the contents), and it will still work the same way. It's a standalone executable!
Next time, we'll implement more functionality for our command line (like
remove) and make it a truly usable package manager!
EDIT: Part 3 is here!
@firefish 1. Look at
Eros. It's a language I'm making, and I need a package manager for it. I started working on
crater and thought "Hey, I bet other people would want to know how to do this".
2. What other language would I use? C? Go is like the language for HTTP when it comes to compiled languages, so it makes sense to use it.