Learn to Code via Tutorials on Repl.it!

← Back to all posts
How to make a package manager (for a custom language!) — Part 2: The Command-Line Tool
fuzzyastrocat

This is part 2 of a series on how to build a package manager. If you haven't gone through part 1, check it out here.

Part 1 now contains all 3 parts, so go there.

Recap

Let's quickly go over what we've done so far.

  • We've built a CDN, which stores package data and will deliver it to our tool.
  • Our CDN responds to GET requests on the url /package/<packagename>/<version>/<file>. Any invalid fields will result in a 404 error and a message such as "This package does not exist".

So, our task at hand: build a command-line tool which gets the package data from the CDN and places it in a nice location. For this tutorial we'll use ./pkg_data/.

Let's get started! (The example repl can be found at https://repl.it/@fuzzyastrocat/cmd-line. As with before, feel free to follow along if you like.)

Language Choice

For this section, I'm going to use Go. I'm using Go because of two reasons:

  1. Go works nicely for HTTP things
  2. Go is compiled

This second one is especially important here — we need our tool to be compiled since we want to be able to create an executable from it. So, as with last time, it's okay if you want to use a different language, but consider carefully based on the two criteria above.

Implementation of the Command-Line Tool

First things first, we need to set up our repl.it workspace. Make a new Go project. Inside our new Go project, create a file called .replit. This is a magical special file that repl.it looks at when you click the run button. In it, put the following:

This tells repl.it that our repl's main language is Go and that when we click the run button it should execute go build -o packager -ldflags \"-s -w\" main.go. (The ldflags just tell the go compiler not to generate version or debug info embedded in the executable. We don't really need to worry about that info since this will become a standalone executable.) -o packager tells Go that we want to output our new executable to ./packager. DO NOT remove this or use -o main: repl will hide an executable called ./main by default, so you won't realize that the program has compiled!

The .replit file overrides repl's default behavior, which is to run the go file. Instead, we simply build it, meaning that an executable (./packager) is generated. We can then run this executable by typing ./packager in the shell (on the right). When you are done with your packager, this will be the actual tool — that way, you can run it without any need for the go compiler.

Okay, now that we've set up our project let's start implementing our interface. The first thing we need to do is check what package the user wants. When you run the executable like ./packager install some_package, ./packager, install and some_package will get given to Go in the form of an array called os.Args. To use it, we need to import os. Let's make a quick program that just echoes the arguments we give it:

Now, click the Run button of your repl. Wait for it to finish running — you should see a file named "packager" get generated. Once it's done building, type ./packager these are some arguments into the shell on the left — the program should print those arguments back to you.

"Wait, why do we ignore the first argument?" Remember that the first argument will always be ./packager, and we don't care about the name of the program.

Okay, let's do something more useful with those arguments. We'll expect arguments of the following form:

We'll start by making the only option install — we include this only for later extensibility. Let's simply start by printing out the available versions of a package:

Obviously, replace "your-cdn-url.repl.co" with your actual cdn url from the last tutorial.
Run the repl, and try ./packager install foo. It should respond with "That package does not exist!". Try again with ./packager install example (assuming that you followed the previous tutorial), and it should print out ["1.0.0"].

Okay, now we need to figure out how to parse this array. We'll make use of Go's encoding/json package, since it parses JSON arrays nicely:

Note that versions is a []byte, which is what json.Unmarshal takes as an argument, so we don't do any string() conversion. If all goes correctly, ./packager install example should now print out "Versions: [1.0.0]".

Now we need to figure out how to find the latest version. We'll assume that all versions all well-formed sequences of 3 numbers separated by a . (things like 1.2.3 and 2.10.0 are fine but 0..0 and 0.5 are not), which makes writing a compare function easier:

Now, we just need to iterate through the array of versions to find the latest (we assume there is at least one version):

So now we can tie it all together to find the files in the package:

Once again, replace your-cdn-url.repl.co with your actual cdn url. If running ./packager install example prints out ["main.lang"] (or whatever your example filename was), great! We can now move on to actually (finally!) fetching the files.

First, we need to parse the array. Then, for each file in the array, we need to fetch (http GET) the contents of the file, which we can then save in a local directory. First though, we'll just print the contents of each file. Replacing fmt.Println(string(files)), here is the relevant code to complete this task:

(Replace your-cdn-url.repl.co as needed.) I won't explain this in detail because there's really no new code here. We're just doing what we've done before, except extended slightly.

So... try it out! Last tutorial, I put Test file! as the contents of main.lang. Therefore, I should expect Test file! to be printed to the console (which it was). If your program does this correctly, then we can move on to the final section: doing something useful with this file data.

First of all, we need to create a folder called ./pkg_data if it doesn't exist. This is easy with os.Mkdir:

(This code goes at the top of main(), you'll see it in the completed code linked below.) 0755 is a permissions number, it means that you'll be able to do anything with the folder and other users can only read it and enter it (but not modify it).

After we get the packagename from os.Args, we need to create a folder for the package. We can do that in the same way as above, with os.Mkdir:

Now we can finally write the files into our local package location. We can write to and make a file with os.Create:

(This replaces fmt.Println(string(contents)).) Okay, the big moment has come: try building and running ./packager install example. If all goes well, you should have a file main.lang inside ./moduledata/example/, with the contents you put in your CDN!

Completed code can be found here. Obviously, replace all occurrences of pkg-cdn.fuzzyastrocat.repl.co with your CDN url.

Conclusion

Wow, that was a lot of work for an utterly trivial result! But now, we do have a minimal package manager which can fetch and "clone" packages from our CDN. Play around with the CDN — try adding more files to the example package and use your command line tool to fetch them. Just remember to re-run your CDN (not the command line tool!) every time you add or change hosted package files. (Note that right now our install option both installs and/or updates, we'll separate out that duty later.)

The nice thing about using a compiled language is that you can take the ./packager executable, put it anywhere (within the same type of system, obviously a Windows-generated executable won't work on a Mac), rename it, do whatever (without modifying the contents), and it will still work the same way. It's a standalone executable!

Next time, we'll implement more functionality for our command line (like update and remove) and make it a truly usable package manager!

EDIT: Part 3 is here!

Voters
programmeruser
VanceBakalov
Viper2211
Highwayman
HahaYes
DynamicSquid
fuzzyastrocat
Comments
hotnewtop
firefish

oh, gosh JUST STOP THE BLASTED TORTURE, you idea stealer

fuzzyastrocat

@firefish What did I do lol

firefish

@fuzzyastrocat are you kidding me, you didn't steal this idea, and happen to come up with the exact same idea for something in exactly the same programming language

fuzzyastrocat

@firefish 1. Look at Eros. It's a language I'm making, and I need a package manager for it. I started working on crater and thought "Hey, I bet other people would want to know how to do this".
2. What other language would I use? C? Go is like the language for HTTP when it comes to compiled languages, so it makes sense to use it.

fuzzyastrocat

@firefish Also I'm sorry that I've made you hate me, I like anyone who thinks AT&T is better than NASM :D

firefish

@fuzzyastrocat yeah but the code is literally someone with bit more experience in golang rewrote it, so admit what you did

fuzzyastrocat

@firefish How do you know how much experience I've had with golang? Also in the next part there's some kinda trashy code that could really be done better

firefish

@fuzzyastrocat Well dusk us literaly the first program in golang ever

fuzzyastrocat

@firefish Oh I get it, you're saying that my code is like yours rewritten but better. Heh I'm bad at Go, I don't think this is good code

HahaYes

@firefish oof you are having competition. Are you getting fired by squid soon XD

fuzzyastrocat

@HahaYes Whoops, didn't intend this to put firefish out of his position lol

DynamicSquid

These tutorials will put johnstev firefish out of work lol

fuzzyastrocat

@DynamicSquid Pardon my ignorance... Is johnstev firefish?

HahaYes

@DynamicSquid lol johnstev oof

DynamicSquid

@HahaYes ah whoops I meant firefish lol. sorry that's just his github username

firefish

@DynamicSquid aha not anymore