Quickly create web services using OpenAPI

OpenAPI is a documentation format created by Swagger to document how a web service works. It defines the paths, the methods and parameters. Let’s know how to quickly create web services using OpenAPI

Why I use it?

Usually documentation is one of the tasks that developers don’t like because it can be boring, hard, and take time. In this case it is really quick and it can be a way to increase your productivity, because this documentation can be used to generate.

Starting with OpenAPI

The OpenAPI format can be generated and viewed with a lot of tools, but my favorite is Apicurio, a free tool from Redhat. A great feature of it is that it can be collaborative

Apicurio studio
How Apicurio looks like

The Apicurio screen is splited on 4 parts

  • Paths: In this part you can see the paths the application uses, you can also click on it to view more details.
  • Data Types: The data types that the application will use and the relation with other data types.
  • Responses : The diferent types of responses of the application
  • Design: The content of this part can change to edit the details of the previous parts
Example of the configuration of a path

Once we have our OpenAPI defined , here starts the magic. We can generate a template code for our applications. To do this you can use this application https://github.com/OpenAPITools/openapi-generator.

Before generation, the code you will need to download is the OpenAPI definition from the Apicurio web

OpenAPI to Flask

If you are used toPython Flask is a great micro framework to create web applications.

openapi-generator-clo generate -g python-flask -i OpenApi.json -o ./code

OpenAPI to Angular

OpenApi can also help you if you are developing a frontend application. You can generate the service of the API using a simple command and it will generate you the code of the service

openapi-generator-cli generate -g typescript-angular -i OpenApi.json -o ./code

ZSH configuration tricks

Zsh is a terminal for UNIX systems that can be tuned to help you do more work with less effort. In this article I will show you some things that are useful to me.

Usually all the Linux distributions used bash as a default terminal, but there are a lot of other terminals like zsh, csh,tcsh. Most of the time I used Linux I used the default terminal but I tried other terminals and discovered that I like zsh more than bash. You can try another terminal without configuring it as your default terminal, just install the zsh package and execute the zsh command.

Oh my zsh

On this article I used a framework for zsh called oh my zsh that extends the default zsh adding plugins and themes. You can easily install with one command:

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Configuration

Plugins

When the zsh starts the terminal, reads the .zshrc file located on the root of your home directory.

The first parameter of configuration that I use is the oh my zsh plugins:

plugins=(
  git
  python
  pip
  virtualenvwrapper
)

This enables the following plugins:

  • git: This is my favorite one,it adds a lot of alias to manage the git, you can check them here. Also , if you are on a git directory indicates if there are unstashed files.
  • python: Like git it adds some useful alias
  • pip: Adds cache on the available pip packages
  • virtualenvwrapper: Auto activates the virtualenv when you cd in a directory that have the same name as an existing virtualenv

Theme

On oh my zsh you can select the theme of your terminal. In my case my favorite is agnoster. You can set it like this:

ZSH_THEME="agnoster"

If you wan to explore the available themes you can find the list here

Alias

On zsh, like in other terminals you can set an alias to shortcut a command. Mine are these:

alias p="ipython3"
alias s="ssh"

These are two applications that I use very often so in this way I can do “s 192.168.1.22” and enter to another machine.

In the case of python,it is because I use the ipython to do quick calculations or test code

CDPATH

When using WSL navigating to different folders can be tedious and repetitive because you are always typing /mnt/c… . To avoid repeating this you can configure the CDPATH variable

CDPATH is a variable that you can use to indicate to the shell where to check when you change of directory. I started using a hash, that is like an alias of a folder, but after I discovered CDPATH that is far better. I have it configured like this:

typeset -path cdpath
setopt auto_cd
cdpath=(/mnt/c/)

On a Linux machine I use the CDPATH with the source directory to move quickly to my source code. You can add multiple directories separated by spaces

Configure sentry on a Ionic Angular app

Sentry is a great tool tool to track errors in our applications, when an error occurs, it is sent to the server, storing it’s trackback and you will receive a notification email. In this article we will learn how to configure sentry on an Ionic Angular Application. On other articles we have seen how to use it with Python.

To explain this, we will start by creating a simple Ionic Application

Sentry with Ionic
Sentry logo

First we create the example application:

ionic start example --type angular tags

Configuration

After this we need to install the npm package

npm install --save @sentry/angular @sentry/tracing

On the main.ts we configure the sentry DSN like this:

import { enableProdMode } from '@angular/core';
import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';

import { AppModule } from './app/app.module';
import { environment } from './environments/environment';

import * as Sentry from '@sentry/angular';
import { Integrations } from '@sentry/tracing';


Sentry.init({
  dsn: 'your sentry dsn',
  integrations: [
    new Integrations.BrowserTracing({
      tracingOrigins: ['localhost'],
      routingInstrumentation: Sentry.routingInstrumentation,
    }),
  ],

  // We recommend adjusting this value in production, or using tracesSampler
  // for finer control
  tracesSampleRate: 1.0,
});


if (environment.production) {
  enableProdMode();
}

platformBrowserDynamic()
  .bootstrapModule(AppModule)
  .catch(err => console.error(err));

Add the ErrorHandler on the providers, on app.module.ts

{
    provide: ErrorHandler,
    useValue: Sentry.createErrorHandler({
        showDialog: false, //You can enable/dissable the report dialog 
    }),
},
{
    provide: Sentry.TraceService,
    deps: [Router],
},
{
    provide: APP_INITIALIZER,
    useFactory: () => () => {},
    deps: [Sentry.TraceService],
    multi: true,
},

Now, with this your application, it is ready to report to the Sentry server. You can test it with calling an undefined function:

undefinedFunction();

If you want to check extra information about this you can check the official documentation

How to have an updated OpenStreetMap database

OpenStreetMap is a crowdsourced database which has editions every second, due this the database that you downloaded a month ago is outdated. To solve this problem I did this little tutorial.

Initial database load

The first step is to load the file to the Postgres. To do this we will use osm2pgsql tool. You can install it with this command

apt-get install osm2pgsql

After this, we can download the actual database from Geofabrik. In this example we will use the data from Andorra:

wget https://download.geofabrik.de/europe/andorra-latest.osm.pbf

Once we have the osm data we load it to the Postgres

createdb osm #Create the database on Postgres
psql -d osm -c "CREATE EXTENSION postgis" # Creates the postgis extension
psql -d osm -c "CREATE EXTENSION hstore" # Creates the hstore extension
osm2pgsql -d osm -s -C 4096 andorra-latest.osm.pbf --hstore # Loads the data 

We need one more step to initialize the replication. In this case we will initialize the osmosis directory on $HOME/osmosis/

mkdir -p $HOME/osmosis/andorra

osmosis --rrii workingDirectory=$HOME/osmosis/andorra
sed -i 's!baseUrl=https://planet.openstreetmap.org/replication/minute!baseUrl=https://download.geofabrik.de/europe/andorra-updates/!' $HOME/osmosis/andorra/configuration.txt
wget https://download.geofabrik.de/europe/spain-updates/state.txt -O "$HOME/osmosis/andorra/state.txt"

Cron task

After we load the database we need to update the database periodically. To do this on Linux we use cron. First we create a scprit, for example on $HOME/update_db.sh with this commands:

osmosis --read-replication-interval workingDirectory="$HOME/osmosis/andorra" --simplify-change --write-xml-change $HOME/andorra_changes.osm
/usr/local/bin/osm2pgsql --append -s -C 3000 -G --hstore -d osm $HOME/andorra_changes.osm

Remember to add execution permisions to the script

chmod +x $HOME/update_db.sh

To configure the task once a day we can add this line on cron

0 7 * * * $HOME/update_db.sh

Your own OpenStreetMap API

OpenStreetMap (OSM) is a great project , good place where to find GIS information and a good software. Sometimes you will be able to solve a project requirement only with the OSM software but you don’t want to make some information public or this information won’t be relevant to be included in OSM.

With this you will be able to create your own OSM server.

Prepare the code

The first step is to clone the code from the oficial OSM repo here but it must be with the branch docker-compose-take2, we will have to do:

git clone https://github.com/openstreetmap/openstreetmap-website.git

Prepare the data

Now we need the data to load on our database. I recomend to download it from Geofabrik . To do this test we will download an small country like Andorra.

wget https://download.geofabrik.de/europe/andorra-latest.osm.pbf

Prepare the docker

To prepare the docker first we need to create the docker image, this is described on the docker-compose file so run:

docker-compose build
docker-compose up -d

After this, we will load the data on the database

cp config/example.storage.yml config/storage.yml
cp config/docker.database.yml config/database.yml
touch config/settings.local.yml

docker-compose build
docker-compose up -d

docker-compose run --rm web bundle exec rake db:migrate # To prepare the db schema
docker-compose run --rm web osmosis \
        -verbose    \
        --read-pbf andorra-latest.osm.pbf \
        --write-apidb \
            host="db" \
            database="openstreetmap" \
            user="openstreetmap" \
            validateSchemaVersion="no"

Now you can see the web working at http://localhost:3000

Create users

This docker image doesn’t have an email server so when you create an user this will not send the validation email and the users will not be activated. To solve this you need to connect to the database (for example with Dbeaver ) and activate the user with this query:

update users set status='active' where status='pending';

Configure iD editor

With your user you have to go to the “User>Preferences>Oauth configuration” and register a new application:

Name: iD

Application URL: http://localhost:3000/oauth/request_token

After this the on the next page it will give us a token that we will need to add to the file /var/www/config/application.yml on the id_key field.

Once we have this we need to restart the apache server with this command:

/etc/init.d/apache restart

5 Python tricks

This is a short list of less known tricks that I use in Python

Delete repeated elements of a list

An easy way to delete repeated elements from a list is to convert it to a set and then convert to a list. This works because a set can’t have repeated elements.

a = [1,1,2,3]
print(list(set(a)))
[1, 2, 3]

Serve static files with HTTP

The python interpreter has the SimpleHttp module which is very useful. If you run this command on a directory you will be able to access to the files thought http.

python3 -m http.server 8080

Join a list of strings with a string separator

A useful not so known str function it’s the join method. You can use this to join a list with a string

",".join(["1","2","3"])
'1,2,3'

With this you don’t have to control if it’s the last element of the list to add the separator

Use a dictionary to pass parameters to a function

Sometimes you need to send data to functions and you need to adapt the same data to the function parameters, you can do this with the operator ** . This operator passes the values of the dictionary to a function using the key as a parameter name:

def function1(param1):
   print(f"function1:{param1}")

def function2(param2):
   print(f"function2:{param2}")

data ="hi"
data_function1 = {"param1": data}
data_function2 = {"param2": data}
function1(**data_function1)
function2(**data_function2)

Make ifs more readables

On of the zen of python is “Readability counts.” . This example shows a correct way to check None and False values on an if

# Wrong way
value = None
if value is None:
   print("No value")

# Wrong way
value = False
if value == False:
   print("Value False)

# Correct way
if not value:
   print("value false or None)

SSH tunnel

SSH is a great tool to open remote secure shells, this is widely used but a less know option SSH has is that can be used to acces the network of the remote server.

Usually if you work with VPS , by security you only open the ports essential, but imagine that you have to access a management web of the server that only can be accessed from loacalhost.

You can solve this with ssh tunneling

ssh remoteserver -L 8090:localhost:8080

This can be a bit confusing but let me explain the parts:

  • ssh remoteserver : Usual ssh remote shell
  • -L : Specifies port forwarding
  • 8089: Local port, the port where your browser will connect
  • localhost: Remote connection, ip address where the remote ssh will connect
  • 8080: Remote port where the remote ssh will connect

If you try this you will notice that it opens a remote shell, if you don’t need it you can additionally add the -N argument to avoid it

Sentry

Sentry is another service you don’t know that you need it until you try it.Basically Sentry is a service that catches the exceptions in your code and sends it to a web service to be analyzed, also sends you an email to notify you.

Sentry server is an opensource server that can be installed on your own server, also sentry can work as a SaS and you can use it at https://sentry.io/welcome/

Sentry can work with multiple languages from Python, to JS or Go, you can check all the supported languages here https://sentry.io/platforms/

Configuring sentry with python

First of all you need to create a project like this:

If you use some framework you can specify

Once you created the project you will get an screen like this

After this you will need to install the sentry package with this command:

pip install --upgrade sentry-sdk

You will need to add this on the package, if you use a setyp.py add it on the install_requires key or if you use it on a requirements.txt file you need to add it.

Now we need to initialize the sentry code. We do it adding this code on the entry point of the code.

import sentry_sdk
sentry_sdk.init(
    "!Place here your project DSN"
    traces_sample_rate=1.0
)

With this we have our script configured to catch all the exceptions with Sentry.

Extra usage

We can make our code notify an exception manually with sentry with this code:

from sentry_sdk import capture_exception
try:
    a=1/0
catch Exception as e:
    capture_exception(e)

Also can notify suspicious data without sending an exception with this code:

from sentry_sdk import capture_message
capture_message("Somthing strange")

DBeaver

DBeaver (https://dbeaver.io/download/) is a great gui tool that allows us to connect and query multiple databases (Postgres,Mysql,Oracle…) from multiple operating systems. DBeaver is free but also has a paid version with extra functionalities

Configuring a connection to Postgres

You can create a new connection going to “File>New>DBeaver>Database connection” or clicking on the socket icon. After this we will see a screen like this:

Screen to select the connection driver

After this we can fill the connection information

My recommendation is to test the connection before finishing

Trick

If you want to use this connection to access different databases you can specify this on the PostgreSQL tab, and enable the “Show all databases” option

You can configure more databases like MySQL or Oracle

Configuring a connection with a SSH tunnel

A cool feature of DBeaver is that it can connect to a database that is on a remote machine without having to expose the service to the whole Internet.

To do this you can go to the tab of “SSH” and configure the

Once you fill the host,user and password you can test the connection and finish

Other functionalities

Viewing the database schema

With DBeaver you can see the database diagram using this button or you can see the relations of a single table.

The result is something like this:

Viewing gis data

A cool feature of DBeaver is the integration with Postgis, so if you double click over a geometry field you will be able to preview the feature in a map

Git ignore

Git ignore is a special file in git version control, its specified by the file .gitignore .

As the name indicates, it’s used to make that git not track specific files. For example when you have application logs, configuration files or compiled files.

Examples

If you want to ignore a, specific directory , let’s call it config:

config/*

To ignore all files with the .log extension

*.log

You also can ignore a, specific file, like passwords.pass

passwords.pass

Another less known option is the exceptions, you can use ! to indicate that you want to track an ignored part of a path. In this example you will ignore all except /foo/bar

/*
!/foo
/foo/*
!/foo/bar

Tools

Usually when you make a project you use a template depending on the language that you are using.

You can use this tool to generate a language specific gitignore:

https://www.toptal.com/developers/gitignore

Github also haves a repository with gitignore templates for different languages

https://github.com/github/gitignore

« Entrades més antigues

© 2021 Another dev

Tema de Anders NorenAmunt ↑