Archiv

Archiv für März, 2020

SSH Server Upload Speed Limit

26. März 2020 Keine Kommentare

Some daemon processes which handle user connections don’t have an internal mechanism to limit the bandwidth. One popular example is OpenSSH.

In theory SFTP could eat up all the bandwidth available, which is especially bad for home connections, since a fully utilized uplink is sometimes worse than a fully utilized downlink.

In order to apply a speed limit to daemon processes which use fork, tools like trinkle are not an option, but tc can do the job.

The following example configures a 10MBit/s upload speed limit (10,5Mbit/s burst limit) to the SSH server (sport 22) running on eth0.

tc qdisc add dev eth0 root handle 1:0 htb

tc class add dev eth0 parent 1:0 classid 1:1 htb \
 rate 10000kbps ceil 10500kbps prio 0

tc filter add dev eth0 protocol ip parent 1:0 prio 0 \
 u32 match ip sport 22 0xffff flowid 1:1

To watch the applied rules and the status:

tc -s -d class show dev eth0

To remove the applied configuration:

tc qdisc del dev eth0 root

This is completely done using the Kernel API. So it can be done with every service by specifying the source port and doesn’t bring any noticeable performance degradation beside having a speed limit applied.

If you want to limit all the upload traffic on a certain interface there doesn’t need to be an filter rule. This is done using a default.
The following example sets the limit from above to all upload traffic on eth0

tc qdisc add dev eth0 root handle 1:0 htb default 1

tc class add dev eth0 parent 1:0 classid 1:1 htb \
 rate 3500kbps ceil 4000kbps prio 0
KategorienEnglish, Linux

Debian 10 – PHP / Matomo GeoIP2 Installation

24. März 2020 Keine Kommentare

The old GeoIP legacy release (aka. GeoIPCity.dat) has been deprecated and doesn’t receive updates for quit some time. It will get removed from Matomo with the next major release.
If you would like to have accurate location detection in Matomo / Piwik the new GeoIP2 module should be used.

Since Debian Buster doesn’t ship it with it’s php packages, there are some manual steps needed to achieve that.

First of all install the needed library, build-tools and the php-dev package:

apt-get install libmaxminddb-dev php-dev git build-essential

Next step is to clone the MaxMind-DB-Reader-php repository in order to compile the PHP module.

cd /usr/local/src
git clone https://github.com/maxmind/MaxMind-DB-Reader-php
cd MaxMind-DB-Reader-php/ext
phpize
./configure
make && make install

After that the new PHP module needs to be enabled and Apache or PHP (when using FPM) restarted:

echo "extension=maxminddb.so" > /etc/php/7.3/mods-available/maxminddb.ini
phpenmod maxminddb

# The folder /usr/share/GeoIP needs to be added
# to the PHP open_basedir setting in your php.ini

apache2ctl graceful
service php7.3-fpm restart

Now the database GeoIP2-City.mmdb needs to be installed locally. Since the free version is updated once a month, this is done via a download cronjob.

apt-get install python3-requests

mkdir -p /usr/share/GeoIP

cat >> /usr/local/sbin/geoip2_update.py << EOF
#!/usr/bin/env python3
import requests
import datetime
import gzip
import sys

db = '/usr/share/GeoIP/GeoIP2-City.mmdb'
url = 'https://download.db-ip.com/free/'
now = datetime.datetime.now()
date = now.strftime('%Y-%m')

r = requests.get(url + 'dbip-city-lite-{}.mmdb.gz'.format(date))
if r.status_code == requests.codes.ok:
  with open(db, 'wb') as f:
    f.write(gzip.decompress(r.content))
  sys.exit(0)
else:
  print('GeoIP2 download failed: ' + str(r.status_code))
  sys.exit(1)
EOF

chmod 700 /usr/local/sbin/geoip2_update.py

cat >> /etc/cron.d/geoip2-update << EOF
MAILTO=root
0 7 28 * *     root  /usr/local/sbin/geoip2_update.py
EOF

Now the GeoIP2 database gets updated once a month automatically and the script should be executed once to get the current version right now and to test the setup as well.

/usr/local/sbin/geoip2_update.py

In order to use GeoIP2 in Matomo two settings need to be done. First of all a database Symlink needs to be created in order to use the GeoIP2 database in Matomo.

# adjust path to your matomo misc folder
ln -s /usr/share/GeoIP/GeoIP2-City.mmdb /var/www/matomo/misc/

After that the Plugin GeoIp2 needs to be activated. It’s a core package and should be already listed on the Matomo Plugins page in the Admin UI.

Now it should be possible to activate the new module under Geolocation. If your location is listed on the right side, enable the GeoIP2 box and click Save.

That’s it. The results are much more accurate and quicker.

If you don’t use Matomo a PHP GeoIP2 testscript could be used instead.

<?php

use MaxMind\Db\Reader;
$reader = new Reader('/usr/share/GeoIP/GeoIP2-City.mmdb');
$record = $reader->get($_SERVER['REMOTE_ADDR']);

print($record['continent']['names']['en'] . "\n");
print($record['country']['names']['en'] . "\n");
print($record['subdivisions'][0]['names']['en'] . "\n");
print($record['city']['names']['en'] . "\n");

?>

Your continent, country, subdivision and city should be listed after executing the script via the webserver.

KategorienApache - PHP, Debian

Host a Jupyter / Python generated Plotly chart

22. März 2020 Keine Kommentare

One thing I do often is generating charts with Jupyter. Common ways to do that with a Python kernel is to use Matplotlib or Plotly. I’m a Plotly user.

The challenge today was to host a Python generated chart for usage in a website. In this particular case it is a S3 bucket, but it could be any webserver.
The easiest way to achieve that is to use static images like .png or .svg, but since Plotly comes with the feature of being interactive via Plotly.js, it’s not what I wanted to do.

The problem in this case was that nearly all Plotly.js examples do the data preparation or generating part in Javascript. But what if you are a Python / Pandas user and have already done that.
What needs to be done if the data is already there?

The answer is to use the .json export feature in Plotly and import it via Plotly.js.

Basic example directory structure of the webserver document_root:

- index.html
- plot.js
- plots/
-- my_plot.json

index.html

<html>
  <head>
    <meta charset="utf-8"/>
    <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
    <script src="plot.js"></script>
  </head>
  <body>
    <div id="my_plot"></div>
  </body>
</html>

The <div> element(s) is getting referenced by name in plot.js and is actually the location of the chart(s) in the DOM.

plot.js

var plots = ['my_plot']
for (let plot of plots) {
  Plotly.d3.json('plots/' + plot + ".json", function(error, data) {
    if (error) return console.warn(error);
    Plotly.newPlot(plot, data.data, data.layout, {responsive: true});
  });
}

Since the .json document my_plot.json was used inside Jupyterlab it contains all information which are needed to generate a graph. The data and layout just needs to be loaded and re-used with Plotly.js
Responsive mode is turned on, no idea why it’s not default :-)

That’s it for the web part. All that’s missing is a valid my_plot.json file. This can be stored via Jupyter or directly uploaded to S3 in case if you host via S3 or run the Python script with Lambda.

Generate my_plot.json

# Store locally
import plotly.graph_objects as go

animals=['giraffes', 'orangutans', 'monkeys']
fig = go.Figure(data=[
    go.Bar(name='SF Zoo', x=animals, y=[20, 14, 23]),
    go.Bar(name='LA Zoo', x=animals, y=[12, 18, 29])
])
fig.update_layout(barmode='group')
fig.show()
# save
fig.write_json('plots/my_plot.json')
# Alternative upload to S3
import boto3
s3 = boto3.resource('s3', region_name='eu-central-1')
s3_object = s3.Object('my_bucket', 'plots/my_plot.json')
s3_object.put(Body=fig.to_json())

That’s it. Now the interactive Plotly chart get’s displayed in the browser as it was before in Jupyterlab.

KategorienJupyter, Python