site-packages?One of the module resolution locations is a site-packages directory.
If you don’t use a venv, this is basically a global install location for packages on your system.
Modules have a __path__ property with their location on disk:
❯ python
Python 3.11.7 (main, Dec 4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__path__
['/Users/gustavo/code/ghidalgo3.github.io/.venv/lib/python3.11/site-packages/requests']
In this case, I am using a venv so site-packages is local and NOT the sytem python on macOS.
__init__.py?This file is implicitly executed when a module package is imported for the first time.
Have a “frontend” like build and run it on your pyproject.toml file.
For example, run python -m build in the directory where a pyproject.toml file exists to build a wheel, many other CLI arguments available.
More context, I’m used to the .NET model where repos contain several projects, MSBuild understands ProjectReferences and building at the top-level understands the topology of the project graph.
At work, we have repos like PCTasks that contain several python projects and my understanding is that Python doesn’t “understand” this dependency “graph”.
In that, modifying source code in the same repo doesn’t necessarilly mean that it will affect what you want it to affect.
UPDATE: This works the way I want it to work. The trick is to:
pip install -eI confirmed that’s actually what happens in the install script of planetary-computer-tasks.
pip installs?Use pip-tools and compile a requirements.txt.
At $DAYJOB I maintain a lot of Python code, so to make everything transferrable I’m going to create a Python package called archetype which is your archetypical Azure Python REST service.
The only design constraint is that archetype must not do anything interesting!
WYSIWYG, so to speak.
Its source code can be found here, and for my sanity I’ll document a little of its structure here:
scripts directory holds the project scripts according to the scripts-to-rule-them-all pattern.pyproject.toml defines metadata for the project. This file is involved when packaging this into a wheel (a python package) and if I was publishing this on PyPI then this file would drive the metadata that others see about this package.src holds the source code of the project. By default, the build backend Hatchling will use either these files for wheels, and these files for source distributions. This might be very important to understand if I was publishing on PyPI but for deploying services it seems not too important.requirements.txt and dev-requirements.txt are generated by pip-compile using pyproject.toml and are not meant to be edited by hand. They are checked in and modified whenever the dependency set of the project changes.Dockerfile simply copies the code, runs setup to install dependencies and install the package as an editable install, and then runs the project using uvicorn as the server.This is the output I see when running the project:
❯ ./scripts/server
INFO: Will watch for changes in these directories: ['/Users/gustavo/code/ghidalgo3.github.io/k8s']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [39552] using StatReload
Importing archetype module!
INFO: Started server process [39554]
INFO: Waiting for application startup.
INFO: Application startup complete.
Dockerizing this project is pretty straight forward:
FROM python:3.11-bookworm
WORKDIR /app
COPY . /app
RUN scripts/setup
ENTRYPOINT [ "uvicorn", "archetype.main:app", "--host=0.0.0.0"]
Line by line:
python:3.11-bookwork/app/appscripts/setup scripts inside the container. This install our dependencies!uvicorn to run its server.Then I publish this container to a private ACR with docker push -t "gustavo2acr.azurecr.io/archetype:latest". Pretty simple!
Eventually I’ll come to find out that docker files are built for specific ISAs!
When I build images on my Mac, the image is an ARM build, not an x86 build.
If I want the image as is to run on the Kubernetes nodes I created, I would have needed to create a cluster with ARM CPU VMs.
The solution is simply to ask docker to build an x86 image with --platform=linux/amd64 in the docker build call.
I’ll do this in the most verbose way possible first to motivate using Helm.
Verbose mandates tool purity, so kubectl is my only choice.
To authenticate againt the cluster, I simply run:
az aks get-credentials \
--resource-group $(terraform output -raw resource_group_name) \
--name $(terraform output -raw k8s_cluster_name)
Double checking that worked…
❯ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 20h
Cool, I have a configured kubectl.
Building this from the bottom up, my understanding is that I need 3 kubernetes resources:
Deployment objectService objectIngress objectAccording to the docs, a Deployment resource declaratively controls Pods and ReplicaSets. I think since 90% of all kubenetes applications just need the cluster to make sure that N replicas of the application are running, deployments were created to satisfy this common use case.
Here’s our deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: archetype-deployment
labels:
app: archetype
spec:
replicas: 2
selector:
matchLabels:
app: archetype
template:
metadata:
labels:
app: archetype
spec:
containers:
- name: archetype
image: gustavo2acr.azurecr.io/archetype:latest-amd64
ports:
- containerPort: 8000
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Then after applying it:
> kubectl describe deployment
Name: archetype-deployment
Namespace: default
CreationTimestamp: Sun, 25 Feb 2024 12:35:33 -0500
Labels: app=archetype
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=archetype
Replicas: 2 desired | 2 updated | 2 total | 0 available | 2 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=archetype
Containers:
archetype:
Image: gustavo2acr.azurecr.io/archetype:latest-amd64
Port: 8000/TCP
Host Port: 0/TCP
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing True ReplicaSetUpdated
OldReplicaSets: <none>
NewReplicaSet: archetype-deployment-75cc984b68 (2/2 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3m23s deployment-controller Scaled up replica set archetype-deployment-75cc984b68 to 2
I’m cheap, so my cluster only has 1 node and it’s a little 2-core Azure VM. It’s remarkable how many pods are running by default in a cluster doing nothing:
❯ kubectl describe node
Name: aks-default-42366886-vmss000000
Roles: agent
Labels: agentpool=default
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_DS2_v2
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=eastus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.azure.com/agentpool=default
kubernetes.azure.com/cluster=MC_gustavo2-rg_gustavo2-aks-cluster_eastus2
kubernetes.azure.com/consolidated-additional-properties=91882a8a-d34a-11ee-bd45-2a30a704d842
kubernetes.azure.com/kubelet-identity-client-id=abb742dc-7a98-47b0-8d03-cc3d662c02c6
kubernetes.azure.com/mode=system
kubernetes.azure.com/node-image-version=AKSUbuntu-2204gen2containerd-202402.07.0
kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets
kubernetes.azure.com/os-sku=Ubuntu
kubernetes.azure.com/role=agent
kubernetes.azure.com/storageprofile=managed
kubernetes.azure.com/storagetier=Premium_LRS
kubernetes.io/arch=amd64
kubernetes.io/hostname=aks-default-42366886-vmss000000
kubernetes.io/os=linux
kubernetes.io/role=agent
node-role.kubernetes.io/agent=
node.kubernetes.io/instance-type=Standard_DS2_v2
storageprofile=managed
storagetier=Premium_LRS
topology.disk.csi.azure.com/zone=
topology.kubernetes.io/region=eastus2
topology.kubernetes.io/zone=0
Annotations: csi.volume.kubernetes.io/nodeid:
{"disk.csi.azure.com":"aks-default-42366886-vmss000000","file.csi.azure.com":"aks-default-42366886-vmss000000"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 24 Feb 2024 14:28:09 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: aks-default-42366886-vmss000000
AcquireTime: <unset>
RenewTime: Sun, 25 Feb 2024 13:06:03 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
ContainerRuntimeProblem False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 ContainerRuntimeIsUp container runtime service is up
VMEventScheduled False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 NoVMEventScheduled VM has no scheduled event
FrequentDockerRestart False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 NoFrequentDockerRestart docker is functioning properly
ReadonlyFilesystem False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 FilesystemIsNotReadOnly Filesystem is not read-only
FrequentKubeletRestart False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 NoFrequentKubeletRestart kubelet is functioning properly
KubeletProblem False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 KubeletIsUp kubelet service is up
FilesystemCorruptionProblem False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 FilesystemIsOK Filesystem is healthy
KernelDeadlock False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 KernelHasNoDeadlock kernel has no deadlock
FrequentUnregisterNetDevice False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 NoFrequentUnregisterNetDevice node is functioning properly
FrequentContainerdRestart False Sun, 25 Feb 2024 13:02:16 -0500 Sat, 24 Feb 2024 14:29:40 -0500 NoFrequentContainerdRestart containerd is functioning properly
MemoryPressure False Sun, 25 Feb 2024 13:04:21 -0500 Sat, 24 Feb 2024 14:28:09 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 25 Feb 2024 13:04:21 -0500 Sat, 24 Feb 2024 14:28:09 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 25 Feb 2024 13:04:21 -0500 Sat, 24 Feb 2024 14:28:09 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 25 Feb 2024 13:04:21 -0500 Sat, 24 Feb 2024 14:28:10 -0500 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.0.1.4
Hostname: aks-default-42366886-vmss000000
Capacity:
cpu: 2
ephemeral-storage: 129886128Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7097684Ki
pods: 30
Allocatable:
cpu: 1900m
ephemeral-storage: 119703055367
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4652372Ki
pods: 30
System Info:
Machine ID: 91d02b7297d74d2290142148194c2bec
System UUID: e0c121d3-1932-4eb1-b50f-fbec6960eccd
Boot ID: 36431874-31af-4b99-84d5-11c14da820a8
Kernel Version: 5.15.0-1054-azure
OS Image: Ubuntu 22.04.3 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.7.7-1
Kubelet Version: v1.27.9
Kube-Proxy Version: v1.27.9
ProviderID: azure:///subscriptions/394dccd4-6f06-4003-8c9a-671ba0d665c7/resourceGroups/mc_gustavo2-rg_gustavo2-aks-cluster_eastus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-42366886-vmss/virtualMachines/0
Non-terminated Pods: (16 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
app-routing-system nginx-5d4cbcf56b-qwkn4 500m (26%) 0 (0%) 127Mi (2%) 0 (0%) 76m
app-routing-system nginx-5d4cbcf56b-rkrx4 500m (26%) 0 (0%) 127Mi (2%) 0 (0%) 77m
default archetype-deployment-69945d5cf9-klrff 0 (0%) 0 (0%) 0 (0%) 0 (0%) 109s
default archetype-deployment-69945d5cf9-wnh7p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m28s
kube-system azure-ip-masq-agent-lbpkh 100m (5%) 500m (26%) 50Mi (1%) 250Mi (5%) 22h
kube-system cloud-node-manager-qbxww 50m (2%) 0 (0%) 50Mi (1%) 512Mi (11%) 22h
kube-system coredns-789789675-nn9qz 100m (5%) 3 (157%) 70Mi (1%) 500Mi (11%) 22h
kube-system coredns-789789675-ss2hf 100m (5%) 3 (157%) 70Mi (1%) 500Mi (11%) 22h
kube-system coredns-autoscaler-649b947bbd-xr74t 20m (1%) 200m (10%) 10Mi (0%) 500Mi (11%) 22h
kube-system csi-azuredisk-node-69clt 30m (1%) 0 (0%) 60Mi (1%) 400Mi (8%) 22h
kube-system csi-azurefile-node-mwcpv 30m (1%) 0 (0%) 60Mi (1%) 600Mi (13%) 22h
kube-system konnectivity-agent-7565b9c9b8-bmjb2 20m (1%) 1 (52%) 20Mi (0%) 1Gi (22%) 22h
kube-system konnectivity-agent-7565b9c9b8-t55pk 20m (1%) 1 (52%) 20Mi (0%) 1Gi (22%) 22h
kube-system kube-proxy-nct72 100m (5%) 0 (0%) 0 (0%) 0 (0%) 22h
kube-system metrics-server-5bd48455f4-6cmbn 50m (2%) 145m (7%) 85Mi (1%) 355Mi (7%) 22h
kube-system metrics-server-5bd48455f4-n8km8 50m (2%) 145m (7%) 85Mi (1%) 355Mi (7%) 22h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1670m (87%) 8990m (473%)
memory 834Mi (18%) 6020Mi (132%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
The interesting bit is that the VM has a CPU capacity of 1900m (about 2 cores) and the current requested CPU consumption is 1670m CPU from all the pods (that I didn’t create BTW!).
The deployment requests to deploy 2 archetype pods that require 500m CPU each so naturally Kubernetes cannot schedule them because 1670m + 2 * 500m > 1900m.
I could create more VMs to increase cluster resourcing but I’m cheap so instead I’ll remove the resource request and limits from the pod spec.
Maybe this is dangerous in production but for my learning purposes I don’t really care about violating resource constraints.
The service definition looks like this:
apiVersion: v1
kind: Service
metadata:
name: archetype-service
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 8000
selector:
app: archetype
Note that the spec.selector.app of the service matches spec.template.metadata.labels.app of the deployment.
This is how kubernetes finds archetype pods to expose as the archetype-service.
If the labels don’t match, then the pods simply aren’t exposed!
❯ kubectl describe service archetype-service
Name: archetype-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=archetype
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 192.168.90.238
IPs: 192.168.90.238
Port: <unset> 80/TCP
TargetPort: 8000/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
ClusterIP services are only available inside the cluster.
That means, a pod within the cluster can communicate with the service either at IP address 192.168.90.238 or at address archetype.
The outside world cannot reach this service!
For that we need to define an Ingress object.
This is where the Azure documentation falls apart.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: archetype-ingress
spec:
ingressClassName: webapprouting.kubernetes.azure.com
rules:
- host: archetype.gustavohidalgo.com
http:
paths:
- backend:
service:
name: archetype-service
port:
number: 80
path: /
pathType: Prefix
Looking at the limitations:
All global Azure DNS zones integrated with the add-on have to be in the same resource group.
Ok that’s kind of a stupid limitation.
I already have a DNS Zone resource that I created long ago for gustavohidalgo.com in a different resource group and I don’t know if I need to or should create a new DNS zone.
Maybe we don’t need to create a DNS zone, because actually this worked:
❯ curl 4.152.12.223/ -H "Host: archetype.gustavohidalgo.com"
{"Hello":"NAMELESS!"}
And the address 4.152.12.223 was allocated either when I enabled the app routing feature or when I created the ingress.
Who knows, AKS definitely doesn’t tell you when new billable resources are created under your nose!
After all of this, I have a dockerized Python application running in a Kubenetes cluster. Cool, but I still feel like AKS is overly complicated.
In their words:
Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures.
Must it?
api-version and 2) the property name matches exactly what the REST API says with no translations to lower_case_underscores. So the problem then becomes: when Terraform creates or updates a resource you don’t know exactly what API version the provider used and therefore you don’t know exactly what request was sent to Azure and therefore you don’t know exactly what your resource actually looks like. This only makes cloud DevOps harder than it needs to be IMO.NO! The Azure Terraform provider also stores secrets in its lock file. It looks like a bearer token, not sure why it does this. Thankfully GitHub alerted me that I had commited a secret here and I immediately went to delete the resources the secrets reference.
HashiCorp tries to prevent this by asking you to use remote state which is just a stepping stone to upselling you to Terraform Cloud. Following the classic tactic of selling you a tool that creates a problem that only they have a solution for, I’ll keep thinking Terraform is over-engineered.
Deeply review your commit before you push them. If you don’t trust yourself or your team, use technology.
Did you know Terraform for AWS has a boatload more effor put into it than Terraform for Azure? Don’t just believe me, compare the AWS provider with the Azure provider.
I’m not saying ARM templates or Bicep files are “good” but at least I am literally just defining an easy to reason about DAG of idempotent REST calls.
Kubernetes is so great and fast moving that all of the old documentation for it is now deprecated. I was reading a book that talked about ReplicationControllers but if you click on that link you see that actually ReplicaSets are the recommended way of doing this. I get it, software changes, I’m not mad about that. I’m mad that I wasn’t told ahead of time to check the publish date and ignore anything that’s 3+ years old 🤣.
You want to run a local kubernetes cluster?
I hope you don’t mind jank because minikube and kind are janky. For starters, you can’t just docker build an image and expect a local kind cluster to use it.
Writing this made me look up that there are 8 ways to do this 🤣.
The minikube team failed here. By default minikube should look for images built by the local docker host and if it can’t find any images then it should look for them in remote container registries. Of course if you had just pushed an image to docker hub or GCP or Azure or AWS or whatever then you would be perfect! How dare I assume that I should be able to use the computer I already paid for and not a cloud service…
Fine, I’ll play your game Kubernetes. Let’s get cloudy.
I installed:
kubectl. It is not necessary but it makes inspecting clusters a little more fun.OK enough tooling, let’s create a cluster!
A k8s cluster is a set of machines (k8s calls them nodes) running applications (losely defined, a pod is an instance of an application) managed by control plane processes which are usually running in dedicated controller nodes and communicate with “agents” that are running on worker node.
In the following image, we see a cluster with 2 nodes, and a control plane node.
When you run kubectl to administer your cluster, you are making requests to kube-api-server in the control plane.
kube-api-server does some validation on your request and stores it in etcd, a kind of simple distributed database for your control plane nodes.
Then the control plane and the kubelets (agents) running on the nodes coordinate to instantiate the right kind and number of pods that the cluster should be running.
Since we’re in Azure land, let’s write a terraform file to deploy an Azure Kubernetes Service cluster.
Basically an AKS cluster is a VM Scale Set with the kubelet and kube-proxy agents configured to talk to a central kubernetes control plane run by the AKS service.
You “own” the worker nodes and Azure “owns” the the control plane nodes.
The documentation puts it like this:
When you create an AKS cluster, a control plane is automatically created and configured. This control plane is provided at no cost as a managed Azure resource abstracted from the user. You only pay for and manage the nodes attached to the AKS cluster.
Ok let’s write this file:
resource "azurerm_kubernetes_cluster" "aks_cluster" {
name = "${var.prefix}-aks-cluster"
location = "eastus2"
resource_group_name = azurerm_resource_group.rg.name
dns_prefix = "${var.prefix}aks"
network_profile {
network_plugin = "azure"
dns_service_ip = "192.168.255.254"
service_cidrs = ["192.168.0.0/16"]
}
default_node_pool {
name = "default"
node_count = 1
vnet_subnet_id = data.azurerm_subnet.node_subnet.id
vm_size = "Standard_DS2_v2"
}
identity {
type = "SystemAssigned"
}
}
What happens when you get this error?
Code="ServiceCidrOverlapExistingSubnetsCidr" Message="The specified service CIDR 10.0.0.0/16 is conflicted with an existing subnet CIDR 10.0.1.0/24" Target="networkProfile.serviceCIDR"
This helpful troubleshooting page doesn’t actually address the problem, but here’s what I understand: there are 2 address spaces at play here, Azure’s and Kubernetes’.
They cannot overlap because otherwise kube-proxy could not configure unambiguous routing rules between pods.
By default (and the error message tells us this but I can’t find this actually documented anywhere within Microsoft’s documenation), AKS clusters take 10.0.0.0/16.
Not exactly true, I did find it mentioned on this page talking about dual-stack IPv4/6 deployments.
Anyway, I want my virtual network to use 10.0.0.0/16 so AKS needs to bend-the-knee to RFC1918 with this network_profile:
network_profile {
network_plugin = "azure"
dns_service_ip = "192.168.255.254"
service_cidrs = [ "192.168.0.0/16" ]
}
Here are some more constraints on these address spaces you oughta know.
Sidebar about this page, when your customers encounter this problem the first solution shouldn’t be to go and delete the virtual network. If someone is deploying nodes into a virtual network they are doing that on purpose. Don’t ask them to go delete it and start over, the first solution should have been explaining how to change the cluster service CIDR range.
Ok that’s enough for one day, I now have an AKS cluster with one VM up and running.
Next time I’ll actually deploy an application to it an explore the world of kubectl, helm, and Ingress controllers.
msbuild as the build system that resolves dependencies, orders projects topologically, compiles your codes, gathers your output files, creates publishable artifacts, etc.
I learned to author msbuild project files quite well at Microsoft, and I got a few opinions out of it.
In this post, I will summarize the best practices I have learned working with msbuild for C# development since 2016.
Words are important, especially with a build system. In one sentence:
msbuildbuilds projects by sequencing targets which execute tasks with properties and item groups as inputs.
Each word deserves a good definition:
msbuild reads and builds.msbuild calls during the build. Tasks can be built-in to msbuild, code that is evaluated at build time, or calls to arbitrary process.A good example is walking through what happens when you build a C# project.
Every file of source code is put into the Compile item group.
The Build target eventually calls the Csc task (that’s the C# compiler task) and passed every file as an argument to the task.
The property Configuration is also passed to the Csc task to control release and debug builds.
If you use Visual Studio, you need to use the msbuild.exe that ships with the version of Visual Studio you are using.
If you don’t match msbuild.exe versions with Visual Studio versions, undefined things will happen when someone or something else tries to build the project on a different machine.
To guarantee this, you need to use the Developer Command Prompt (or Developer PowerShell) for the version of Visual Studio you use.
Calling that program will put the right msbuild.exe on your PATH, along with a host of other Visual Studio tools..
If you use the dotnet CLI, msbuild is implicitly invoked when you call dotnet build, but you can unleash the beast by calling dotnet msbuild instead.
That command will call the appropriate version of msbuild that ships with the .NET SDK version you are using and forward any command line arguments to msbuild.
Do one of those two and you should not have any issues.
It is important to ensure that developer machines and CI/CD machines all use the same version of the .NET SDK.
That can be controlled by using a global.json file.
The full documentation for global.json can be found here, but basically I always roll with something like this:
{
"sdk": {
"version": "7.0.102",
"rollForward": "latestFeature"
}
}
Directory.Build.propsSince msbuild operates on XML files, you can re-use build settings by Import-ing XML files like this.
For along time, this was the only way to organize and re-use build configuration.
In newer versions of msbuild, a project’s build will search the file system for a file named Directory.Build.props and automatically import that file before building the project.
This means that properties and item groups defined this file will be available to any project that needs to know about them.
The full documentation can be found here, but this is what I roll with:
<Project>
<PropertyGroup>
<Platform Condition=" '$(Platform)' == '' ">x64</Platform>
<Configuration Condition=" '$(DOTNET_WATCH)' == '1' ">Debug</Configuration>
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
<TreatWarningsAsErrors Condition=" '$(Configuration)' == 'Release' ">true</TreatWarningsAsErrors>
</PropertyGroup>
</Project>
Here’s another good Directory.Build.props that I like to reference.
Directory.Build.targetsThe sister file to Directory.Build.props should define target definitions or redefinitions because it is imported after a project is evaluated.
It is highly likely that you will never need write a custom targets.
With any luck you will never need to understand why or actually write a Directory.Build.targets, but in case you do please read through this, familiarize yourself with the /pp command-line flag to msbuild and use your favorite text editor to examine massive XML files :).
Here is an example one I had to write last year:
<Project>
<!--
[REDACTED] uses packages.config instead of PackageReference and they have
static relative paths in their project files to their assembly references.
Since we cannot modify their project files, we will tell MSBuild to rewrite
the HintPath attribute at build time!
-->
<UsingTask
TaskName="PackageFolderRedirect"
TaskFactory="CodeTaskFactory"
AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.Core.dll" >
<ParameterGroup>
<References ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true"/>
</ParameterGroup>
<Task>
<Using Namespace="System"/>
<Using Namespace="System.IO"/>
<Code Type="Fragment" Language="cs">
<![CDATA[
if (References.Length > 0)
{
for (int i = 0; i < References.Length; i++)
{
ITaskItem item = References[i];
string path = item.GetMetadata("HintPath");
if (!string.IsNullOrWhiteSpace(path) && !File.Exists(path))
{
string newPath = "..\\" + path;
Log.LogMessage(MessageImportance.High, "Redirecting HintPath to " + newPath);
References[i].SetMetadata("HintPath", newPath);
}
else
{
Log.LogMessage(MessageImportance.Low, "Valid HintPath");
}
}
}
]]>
</Code>
</Task>
</UsingTask>
<Target
Name="FixHintPath"
Condition="Exists('$(MSBuildProjectDirectory)/packages.config') == 'true'"
BeforeTargets="ResolveAssemblyReferences" >
<PackageFolderRedirect References="@(Reference)" />
</Target>
</Project>
Code organization and architecture are imporant things all software engineers should think about.
I’ve seen code bases where well-intentioned developers used msbuild projects to structure code, and this can be problematic if taken too far.
For example, the first team I joined essentially shipped code out of one repo to 4 locations:
This means there were at 4 msbuild projects (one for each deployment target) and at least 1 “common” project that housed the shared code.
This N + 1 arrangement is the simplest possible architecture for projects, that is N projects that control deployment specific settings and 1 common library that produces a library assembly.
There is the case where you have no shared assemblies, then you only have one project.
In reality, that team had at least 20 intermediate projects forming their own little internal dependency graph within the repo. There are many downsides to this including:
That’s it for now, I’ll come back here and update these as I collect more nuggets of best practices.
]]>I am reading through Crafting Interpreters by Robert Nystom and one of the first exercises asks you to write and debug a doubly-linked list in C. Regretfully I confess I never actually learned how to use GDB, ever, and on macOS I need to use the LLVM toolchain which means using LLDB instead of GDB. In this post I will write using LLDB to debug a simple C program.
Throughout this post, I will reference a simple implementation of a doubly-linked list in C. Here is the single-file C program I will be debugging:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct LinkedListNode {
struct LinkedListNode* next;
struct LinkedListNode* prev;
char* value;
} LinkedListNode;
LinkedListNode* initList(char* value) {
LinkedListNode* head = (LinkedListNode*)malloc(sizeof(LinkedListNode));
head->value = value;
head->next = NULL;
head->prev = NULL;
return head;
}
int append(LinkedListNode* list, char* new) {
LinkedListNode* current = list;
int size = 1;
while (current->next != NULL) {
current = current->next;
size++;
}
LinkedListNode* tail = initList(new);
tail->prev = current;
current->next = tail;
return size++;
}
void printList(LinkedListNode* list) {
LinkedListNode* current = list;
for(int index = 0; current != NULL; current = current->next, index++) {
printf("[%d]: %s\n", index, current->value);
}
}
LinkedListNode* find(LinkedListNode* list, char* value) {
LinkedListNode* current = list;
while (current != NULL && strcmp(current->value, value) != 0) {
// printf("Comparing %s and %s\n", current->value, value);
current = current->next;
}
if (current != NULL) {
return current;
}
return NULL;
}
LinkedListNode* delete(LinkedListNode* list, char* value) {
LinkedListNode* toDelete = find(list, value);
if (toDelete != NULL) {
if (toDelete->prev != NULL) {
toDelete->prev->next = toDelete->next;
}
if (toDelete->next != NULL) {
toDelete->next->prev = toDelete->prev;
}
}
toDelete->prev = NULL;
toDelete->next = NULL;
return toDelete;
}
int main(int argc, char* argv[]) {
LinkedListNode* list = initList("A");
append(list, "B");
append(list, "C");
append(list, "D");
printList(list);
printf("\n");
LinkedListNode* c = find(list, "C");
printList(c);
printf("\n");
delete(list, "B");
printList(list);
}
And the Makefile that builds this:
# Thanks to Job Vranish (https://spin.atomicobject.com/2016/08/26/makefile-c-projects/)
TARGET_EXEC := llist
BUILD_DIR := ./build
SRC_DIRS := ./src
# Find all the C and C++ files we want to compile
# Note the single quotes around the * expressions. Make will incorrectly expand these otherwise.
SRCS := $(shell find $(SRC_DIRS) -name '*.cpp' -or -name '*.c' -or -name '*.s')
# String substitution for every C/C++ file.
# As an example, hello.cpp turns into ./build/hello.cpp.o
OBJS := $(SRCS:%=$(BUILD_DIR)/%.o)
# String substitution (suffix version without %).
# As an example, ./build/hello.cpp.o turns into ./build/hello.cpp.d
DEPS := $(OBJS:.o=.d)
# Every folder in ./src will need to be passed to GCC so that it can find header files
INC_DIRS := $(shell find $(SRC_DIRS) -type d)
# Add a prefix to INC_DIRS. So moduleA would become -ImoduleA. GCC understands this -I flag
INC_FLAGS := $(addprefix -I,$(INC_DIRS))
# The -MMD and -MP flags together generate Makefiles for us!
# These files will have .d instead of .o as the output.
CPPFLAGS := $(INC_FLAGS) -MMD -MP
CFLAGS := -g
# The final build step.
$(BUILD_DIR)/$(TARGET_EXEC): $(OBJS)
$(CC) $(OBJS) -o $@ $(LDFLAGS)
# Build step for C source
$(BUILD_DIR)/%.c.o: %.c
mkdir -p $(dir $@)
$(CC) $(CPPFLAGS) $(CFLAGS) -c $< -o $@
# Build step for C++ source
$(BUILD_DIR)/%.cpp.o: %.cpp
mkdir -p $(dir $@)
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@
.PHONY: clean
clean:
rm -r $(BUILD_DIR)
# Include the .d makefiles. The - at the front suppresses the errors of missing
# Makefiles. Initially, all the .d files will be missing, and we don't want those
# errors to show up.
-include $(DEPS)
Your program must be compiled with debug symbols to fully work with LLDB For Apple’s
ccC compiler, debug symbols generation uses the-gflag passed to the compiler.
Starting simple, LLDB can be invoked by passing it an executable file.
Compilation of our sample code produces a llist executable, so to debug it you simply invoke lldb llist
This launches the debugger and drops you in a REPL prompt, waiting for a command.
The simplest thing to do is to run the program, which can be done by sending the run command:
> lldb llist
(lldb) target create "llist"
Current executable set to '/Users/gustavo/code/craftinginterpreters/prelude/build/llist' (x86_64).
(lldb) run
Process 7890 launched: '/Users/gustavo/code/craftinginterpreters/prelude/build/llist' (x86_64)
[0]: A
[1]: B
[2]: C
[3]: D
[0]: C
[1]: D
[0]: A
[1]: C
[2]: D
Process 7890 exited with status = 0 (0x00000000)
Here we can see that the printf statements from the C program work, and the process exits with the expected 0 status code.
You can run the program again with run, and for a full list of commands you can run help at the debugger REPL.
Pretty neat!
LLDB and GDB both support running commands identified by an unambiguous prefix match.
This means that instead of typing run<Enter> you can just type r<Enter> (or ru<Enter>) because there is no other command that begins with r.
While great for experienced debuggers, use of the terse commands for beginners just causes them to have to learn shortcuts at the same time as learning to use the debugger.
In this post I will use the full command names, but just know that you can always use the shorter names.
Because we passed the -g flag to cc to generate debug symbols, LLDB has full knowledge of the source code of our executable.
Let’s set a breakpoint right at the start of main then run the code:
> lldb llist
(lldb) target create "llist"
Current executable set to '/Users/gustavo/code/craftinginterpreters/prelude/build/llist' (x86_64).
(lldb) breakpoint set --name main
Breakpoint 1: where = llist`main + 15 at llist.c:67:28, address = 0x0000000100003ebf
(lldb) run
Process 2805 launched: '/Users/gustavo/code/craftinginterpreters/prelude/build/llist' (x86_64)
Process 2805 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x0000000100003ebf llist`main(argc=1, argv=0x00007ff7bfeff4b0) at llist.c:67:28
64 }
65
66 int main(int argc, char* argv[]) {
-> 67 LinkedListNode* list = initList("A");
68 append(list, "B");
69 append(list, "C");
70 append(list, "D");
Target 0: (llist) stopped.
(lldb)
Let’s break that down:
lldb llistmain with breakpoint set --name mainrunllist.c line 67 with a source code print out.Neat!
A debugger should allow you to inspect the values of local variables, and you can do that with var.
Within main, we should see local variables for the signature of main and the 2 LinkedListNode* declared inside:
(lldb) var
(int) argc = 1
(char **) argv = 0x00007ff7bfeff4b0
(LinkedListNode *) list = 0x000000010001a883
(LinkedListNode *) c = 0x00007ff7bfeff370
A debugger should allow you to step through your code line by line, and there are 2 commands to know:
step: Executes the next thing, stepping into calls.next: Executes the next thing, stepping over calls.Our breakpoint at main stopped right at a call to initList.
Let’s step inside initList with step:
(lldb) step
Process 2805 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x0000000100003c5c llist`initList(value="A") at llist.c:12:45
9 } LinkedListNode;
10
11 LinkedListNode* initList(char* value) {
-> 12 LinkedListNode* head = (LinkedListNode*)malloc(sizeof(LinkedListNode));
13 head->value = value;
14 head->next = NULL;
15 head->prev = NULL;
Target 0: (llist) stopped.
Now we’re inside of initList and we could set more breakpoints or inspect locals.
One final basic command, finish.
finish allows the current stack frame to run to completion, and then breaks when program execution returns to the caller frame.
This is a good command to run whenever you want to just finish the current function call and return to debugging your caller.
If we call finish while we’re debugging initList, we’ll go back to debugging main:
(lldb) finish
Process 2805 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (LinkedListNode *) $0 = 0x00000001003044a0
frame #0: 0x0000000100003ecb llist`main(argc=1, argv=0x00007ff7bfeff4b0) at llist.c:67:21
64 }
65
66 int main(int argc, char* argv[]) {
-> 67 LinkedListNode* list = initList("A");
68 append(list, "B");
69 append(list, "C");
70 append(list, "D");
Target 0: (llist) stopped.
Finally, you can always terminate the debugging session with exit.
Think about the experience of using an IDE: you set breakpoints on lines and the breakpoints move as you edit code to approximately stay with the line you originally set the breakpoint on.
You can add several breakpoints, run the program, and the breakpoints persist across runs.
In a CLI debugger, you can achieve the same thing by writing a file with commands that gets run when lldb runs a program.
For example, let’s say you knew you wanted to debug printList, and you wanted to spare having to write breakpoint set --name printList and run every time you invoked lldb.
First, write a file containing the commands you want to run, one command per line:
breakpoint set --name printList
run
Then invoke lldb with the --source (short -s) command:
> lldb llist -s ../debug
(lldb) target create "llist"
Current executable set to '/Users/gustavo/code/craftinginterpreters/prelude/build/llist' (x86_64).
(lldb) command source -s 0 '../debug'
Executing commands in '/Users/gustavo/code/craftinginterpreters/prelude/debug'.
(lldb) breakpoint set --name printList
Breakpoint 1: where = llist`printList + 12 at llist.c:33:31, address = 0x0000000100003d2c
(lldb) run
Process 3648 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x0000000100003d2c llist`printList(list=0x00000001003041c0) at llist.c:33:31
30 }
31
32 void printList(LinkedListNode* list) {
-> 33 LinkedListNode* current = list;
34 for(int index = 0; current != NULL; current = current->next, index++) {
35 printf("[%d]: %s\n", index, current->value);
36 }
Target 0: (llist) stopped.
Process 3648 launched: '/Users/gustavo/code/craftinginterpreters/prelude/build/llist' (x86_64)
This debugger command file therefore allows us to configure the debugger on launch with an arbitrarily complex command list, which means you can script the process of getting your program into the error condition you need to debug 😈.
Don’t confuse
-s|--sourcewith-S|--source-before-file! The latter runs the commands before your target executable is loaded, which means your debugger commands will fail!
This post wraps a simple introduction to lldb, the LLVM equivalent to GCC’s gdb.
We saw how to compile, run, and debug a simple C program with lldb.
Then we saw how to script lldb to automatically set breakpoints.
I am happy with lldb’s documentation and its UX, I will use it to debug my C programs.
https://lldb.llvm.org/index.html
]]>Every linked list needs a good recursive linked_list_node definition with a nice car and cdr erm I mean value and next:
struct LinkedListNode<'a, T> {
value : T,
next : &'a LinkedListNode<'a, T>,
}
It’s obvious to me why next has to be a reference to a LinkedListNode, but the lifetime parameter 'a requires some explaining. That will be explained later.
Now obviously we need to support creating an empty list and adding elements to it one at a time.
pub struct List<'a, T> {
head : &LinkedListNode<'a, T>
}
impl List {
fn new() {
// except Rust doesn't have a null!
List { head : null }
}
}
Now here is where I ran into my first C#-induced headache, because I would have loved to use null to represent a the end of a list.
Rust has no nulls, and some Googling immediately pointed me toward using an Option and a great book doing exactly what I’m trying to do!
In short, we can use Rust enums to create a tagged union(or what F# would call a discriminated union) to define a type that can be one of several values (one of which is Null or Nil) and whose values can themselves be more complex types.
Something like this:
enum ListNode {
Empty,
ListNode(i32, Box<ListNode>)
}
I read that like this: A ListNode is either the ListNode::Empty value, or it is a ListNode(i32, Box<ListNode>) which is an i32 paired with a pointer to a heap-allocated ListNode.
I wouldn’t say that starting to write this post was a waste of time because I discovered a great book to read through. Sometimes starting and failing still results in a victory.
]]>jq?
jq is a wonderful tool for processing JSON in the command line.
The prevalence of JSON in web APIs and data dumps makes jq a very imporant tool for interactively manipulating JSON.
The tutorial is good enough to get you started with basic functionality, but I found that real usage quickly outgrew the scope of the tutorial.
In this post I will show examples of using jq in slightly more complex scenarios than the tutorial.
The data in this post comes from Yelp’s Dataset Challenge, and I will be focusing on the business data file (that is yelp_academic_dataset_business.json if you’re following at home).
Usually I like to look at one row of a dataset to understand the data schema.
The Yelp data contains one entry per line, so we can use head -n 1 argument to get the first line of the data file and inspect the schema.
> head -n1 yelp_academic_dataset_business.json
{"business_id":"6iYb2HFDywm3zjuRg0shjw","name":"Oskar Blues Taproom","address":"921 Pearl St","city":"Boulder","state":"CO","postal_code":"80302","latitude":40.0175444,"longitude":-105.2833481,"stars":4.0,"review_count":86,"is_open":1,"attributes":{"RestaurantsTableService":"True","WiFi":"u'free'","BikeParking":"True","BusinessParking":"{'garage': False, 'street': True, 'validated': False, 'lot': False, 'valet': False}","BusinessAcceptsCreditCards":"True","RestaurantsReservations":"False","WheelchairAccessible":"True","Caters":"True","OutdoorSeating":"True","RestaurantsGoodForGroups":"True","HappyHour":"True","BusinessAcceptsBitcoin":"False","RestaurantsPriceRange2":"2","Ambience":"{'touristy': False, 'hipster': False, 'romantic': False, 'divey': False, 'intimate': False, 'trendy': False, 'upscale': False, 'classy': False, 'casual': True}","HasTV":"True","Alcohol":"'beer_and_wine'","GoodForMeal":"{'dessert': False, 'latenight': False, 'lunch': False, 'dinner': False, 'brunch': False, 'breakfast': False}","DogsAllowed":"False","RestaurantsTakeOut":"True","NoiseLevel":"u'average'","RestaurantsAttire":"'casual'","RestaurantsDelivery":"None"},"categories":"Gastropubs, Food, Beer Gardens, Restaurants, Bars, American (Traditional), Beer Bar, Nightlife, Breweries","hours":{"Monday":"11:0-23:0","Tuesday":"11:0-23:0","Wednesday":"11:0-23:0","Thursday":"11:0-23:0","Friday":"11:0-23:0","Saturday":"11:0-23:0","Sunday":"11:0-23:0"}}
😅 A little hard to read. Let’s pipe that into jq
> head -n1 yelp_academic_dataset_business.json | jq
{
"business_id": "6iYb2HFDywm3zjuRg0shjw",
"name": "Oskar Blues Taproom",
"address": "921 Pearl St",
"city": "Boulder",
"state": "CO",
"postal_code": "80302",
"latitude": 40.0175444,
"longitude": -105.2833481,
"stars": 4,
"review_count": 86,
"is_open": 1,
"attributes": {
"RestaurantsTableService": "True",
"WiFi": "u'free'",
"BikeParking": "True",
"BusinessParking": "{'garage': False, 'street': True, 'validated': False, 'lot': False, 'valet': False}",
"BusinessAcceptsCreditCards": "True",
"RestaurantsReservations": "False",
"WheelchairAccessible": "True",
"Caters": "True",
"OutdoorSeating": "True",
"RestaurantsGoodForGroups": "True",
"HappyHour": "True",
"BusinessAcceptsBitcoin": "False",
"RestaurantsPriceRange2": "2",
"Ambience": "{'touristy': False, 'hipster': False, 'romantic': False, 'divey': False, 'intimate': False, 'trendy': False, 'upscale': False, 'classy': False, 'casual': True}",
"HasTV": "True",
"Alcohol": "'beer_and_wine'",
"GoodForMeal": "{'dessert': False, 'latenight': False, 'lunch': False, 'dinner': False, 'brunch': False, 'breakfast': False}",
"DogsAllowed": "False",
"RestaurantsTakeOut": "True",
"NoiseLevel": "u'average'",
"RestaurantsAttire": "'casual'",
"RestaurantsDelivery": "None"
},
"categories": "Gastropubs, Food, Beer Gardens, Restaurants, Bars, American (Traditional), Beer Bar, Nightlife, Breweries",
"hours": {
"Monday": "11:0-23:0",
"Tuesday": "11:0-23:0",
"Wednesday": "11:0-23:0",
"Thursday": "11:0-23:0",
"Friday": "11:0-23:0",
"Saturday": "11:0-23:0",
"Sunday": "11:0-23:0"
}
}
Much easier to read!
To completely format a file, we can pipe all lines into jq and then redirect the result to a file:
> cat yelp_academic_dataset_business.json | jq > yelp_academic_dataset_business_formatted.json
Caution, this file is not valid JSON! It is only useful for reading through the data in a text editor.
Let’s say we wanted to find all of the restaurants with ratings >= 4 and with more than 5 reviews. We can do it like this:
> cat yelp_academic_dataset_business.json | jq --compact-output 'select(.review_count > 5 and .stars >= 4)' > high_ratings.json
Note that I used
--compact-outputto preserve the 1-object-per-line structure of the original file. Without this option,high_ratings.jsonwould contain nicely formatted JSON.
The syntax is very easy: jq 'select(<boolean expression>)' will filter objects where the boolean expression returns false.
Or sometimes called projections, I will show 2 cases building on the previous filter example.
cat yelp_academic_dataset_business.json | jq --compact-output 'select(.review_count > 5 and .stars >= 4) | .business_id' > high_ratings.json
After the select call, I pipe the resulting object into a property access .business_id.
This produces a file that looks like:
"6iYb2HFDywm3zjuRg0shjw"
"tCbdrRPZA0oiIYSmHG3J0w"
"bvN78flM8NLprQ1a1y5dRg"
"PE9uqAjdw0E4-8mjGl3wVA"
cat yelp_academic_dataset_business.json | jq --compact-output 'select(.review_count > 5 and .stars >= 4) | {business_id}' > high_ratings.json
Note that the syntax | {business_id} is a shortcut for | {"business_id": .business_id}, the latter case can be useful if you want to rename a property or product a new property from different values.
This produces a file that looks like:
{"business_id":"6iYb2HFDywm3zjuRg0shjw"}
{"business_id":"tCbdrRPZA0oiIYSmHG3J0w"}
{"business_id":"bvN78flM8NLprQ1a1y5dRg"}
{"business_id":"PE9uqAjdw0E4-8mjGl3wVA"}
JSON is a very self-documenting format for data, which is great for humans but not very useful for computers. If I wanted to do some basic analysis on this data, the first thing I would do is load it into a spreadsheet program and start generating some graphs. To do that, we need to transform this file into a comma-separated value file or a tab-separated value file.
Unfortunately any command to transform JSON to CSV on anything but the most basic JSON has to make choices about what to do with object and array properties (recursively) so there is no easy answer.
The jq filter we write depends on the source data and how we want the CSV to look like.
Or does it?
Denizens of the internet have graciously provided a jq filter to perform this for arbitrary JSON.
This filter takes advantage of jq’s programming features: that’s right, jq is its own little programming language!
Gaze your eyes on this:
def json2header:
[paths(scalars)];
def json2array($header):
[$header[] as $p | getpath($p)];
# given an array of conformal objects, produce "CSV" rows, with a header row:
def json2csv:
(.[0] | json2header) as $h
| ([$h[]|join(".")], (.[] | json2array($h)))
| @csv ;
# `main`
json2csv
Put this into a file called json2csv.jq and invoke it like this:
jq -rf json2csv.jq yelp_academic_dataset_business.json
Don’t forget to turn the json file into an array of objects first with something like:
> head -n 3 yelp_academic_dataset_business.json | jq -s '.' > high_ratings.json
Big thanks to this and this StackOverflow question.
jq is great!
At work we write a lot of ASP.NET REST controllers.
Some of the controllers have their API documented in the Azure REST API specs in the form of a Swagger specification, and some are consumed internally by a system that requires us to expose a Swagger specification. This is our problem: say we want to write a controller for Widgets that supports CRUD operations. A developer has to:
There are existing tools like NSwag and Swashbuckle that are almost tailor made for this but they assume you’re going to generate perfectly legal OpenAPIv2/3 documents, which I’m not because this internal swagger has some… customizations. We can easily adapt a proper Swagger document into our custom format, ultimately we are trying to generate a swagger document at build time from our source code which we will then transform some more (again, at build time!).
Thankfully NSwag has a component that does almost exactly what we need: NSwag.MSBuild. With this package, we can generate a Swagger document for some or all of the controllers in an assembly, and then we can modify the document to fit our custom Swagger format. Let’s see how it works:
First I generate a small project with:
> dotnet new webapi
At this time, this template contains one controller with a GET method on /WeatherForecast that returns an object with weather forecast data.
Then I add a dependency on NSwag with
> dotnet add package NSwag.AspNetCore
> dotnet add package NSwag.MSBuild
This is my Startup.cs with the call to AddSwaggerDocument, this is necessary for the generator to discover controllers in the assembly:
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
public void ConfigureServices(IServiceCollection services)
{
services.AddControllers();
services.AddSwaggerDocument();
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseHttpsRedirection();
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
}
}
And finally I add a target in the project file to call the Swagger generator executable. Note that I am on a Mac so I have to call the generator with the property for .NET 5.
<Target Name="NSwag" AfterTargets="Build">
<Exec
EnvironmentVariables="ASPNETCORE_ENVIRONMENT=Development"
Command="$(NSwagExe_Net50) aspnetcore2openapi /assembly:$(TargetDir)$(MSBuildProjectName).dll /output:swagger.json" />
</Target>
Some nice build output:
Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
Determining projects to restore...
All projects are up-to-date for restore.
swaggergen -> /Users/gustavo/code/swaggergen/bin/Debug/net5.0/swaggergen.dll
NSwag command line tool for .NET Core Net50, toolchain v13.11.3.0 (NJsonSchema v10.4.4.0 (Newtonsoft.Json v12.0.0.0))
Visit http://NSwag.org for more information.
NSwag bin directory: /Users/gustavo/.nuget/packages/nswag.msbuild/13.11.3/tools/Net50
Code has been successfully written to file.
Duration: 00:00:00.8377024
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:03.55
And finally, a swagger.json file is produced next to our project file.
Let’s take a look at it:
{
"x-generator": "NSwag v13.11.3.0 (NJsonSchema v10.4.4.0 (Newtonsoft.Json v12.0.0.0))",
"swagger": "2.0",
"info": {
"title": "My Title",
"version": "1.0.0"
},
"produces": [
"text/plain",
"application/json",
"text/json"
],
"paths": {
"/WeatherForecast": {
"get": {
"tags": [
"WeatherForecast"
],
"operationId": "WeatherForecast_Get",
"responses": {
"200": {
"x-nullable": false,
"description": "",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/WeatherForecast"
}
}
}
}
}
}
},
"definitions": {
"WeatherForecast": {
"type": "object",
"required": [
"date",
"temperatureC",
"temperatureF"
],
"properties": {
"date": {
"type": "string",
"format": "date-time"
},
"temperatureC": {
"type": "integer",
"format": "int32"
},
"temperatureF": {
"type": "integer",
"format": "int32"
},
"summary": {
"type": "string"
}
}
}
}
}
Pretty good! The generator:
paths for each controller method.For a very simple controller, that Swagger document is a whopping 141 lines long! I will grant that it is a full Swagger document with all of the ceremony of Swagger, but still even if it was only 70 lines that would be a huge change to review. If the author tells me “The swagger is automatically generated from the code”, then I’ll spend exactly 0 seconds looking at it and instead focus on reviewing the code that produced it.
Generating Swagger files from C# code is extremely easy with NSwag. Armed with these tools, you can reduce the amount of time you spend toiling at the Swagger mines.
]]>