Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

May 22 2017

Paul Boddie's Free Software-related blog » English: VGA Signal Generation with the PIC32

It all started after I had designed – and received from fabrication – a circuit board for prototyping cartridges for the Acorn Electron microcomputer. Although some prototyping had already taken place with an existing cartridge, with pins intended for ROM components being routed to drive other things, this board effectively “breaks out” all connections available to a cartridge that has been inserted into the computer’s Plus 1 expansion unit.

Acorn Electron cartridge breakout board

The Acorn Electron cartridge breakout board being used to drive an external circuit

One thing led to another, and soon my brother, David, was interfacing a microcontroller to the Electron in order to act as a peripheral being driven directly by the system’s bus signals. His approach involved having a program that would run and continuously scan the signals for read and write conditions and then interpret the address signals, sending and receiving data on the bus when appropriate.

Having acquired some PIC32 devices out of curiosity, with the idea of potentially interfacing them with the Electron, I finally took the trouble of looking at the datasheet to see whether some of the hard work done by David’s program might be handled by the peripheral hardware in the PIC32. The presence of something called “Parallel Master Port” was particularly interesting.

Operating this function in the somewhat insensitively-named “slave” mode, the device would be able to act like a memory device, with the signalling required by read and write operations mostly being dealt with by the hardware. Software running on the PIC32 would be able to read and write data through this port and be able to get notifications about new data while getting on with doing other things.

So began my journey into PIC32 experimentation, but this article isn’t about any of that, mostly because I put that particular investigation to one side after a degree of experience gave me perhaps a bit too much confidence, and I ended up being distracted by something far more glamorous: generating a video signal using the PIC32!

The Precedents’ Hall of Fame

There are plenty of people who have written up their experiments generating VGA and other video signals with microcontrollers. Here are some interesting examples:

And there are presumably many more pages on the Web with details of people sending pixel data over a cable to a display of some sort, often trying to squeeze every last cycle out of their microcontroller’s instruction processing unit. But, given an awareness of how microcontrollers should be able to take the burden off the programs running on them, employing peripheral hardware to do the grunt work of switching pins on and off at certain frequencies, maybe it would be useful to find examples of projects where such advantages of microcontrollers had been brought to bear on the problem.

In fact, I was already aware of the Maximite “single chip computer” partly through having seen the cloned version of the original being sold by Olimex – something rather resented by the developer of the Maximite for reasons largely rooted in an unfortunate misunderstanding of Free Software licensing on his part – and I was aware that this computer could generate a VGA signal. Indeed, the method used to achieve this had apparently been written up in a textbook for the PIC32 platform, albeit generating a composite video signal using one of the on-chip SPI peripherals. The Colour Maximite uses three SPI channels to generate one red, one green, and one blue channel of colour information, thus supporting eight-colour graphical output.

But I had been made aware of the Parallel Master Port (PMP) and its “master” mode, used to drive LCD panels with eight bits of colour information per pixel (or, using devices with many more pins than those I had acquired, with sixteen bits of colour information per pixel). Would it surely not be possible to generate 256-colour graphical output at the very least?

Information from people trying to use PMP for this purpose was thin on the ground. Indeed, reading again one article that mentioned an abandoned attempt to get PMP working in this way, using the peripheral to emit pixel data for display on a screen instead of a panel, I now see that it actually mentions an essential component of the solution that I finally arrived at. But the author had unfortunately moved away from that successful component in an attempt to get the data to the display at a rate regarded as satisfactory.

Direct Savings

It is one thing to have the means to output data to be sent over a cable to a display. It is another to actually send the data efficiently from the microcontroller. Having contemplated such issues in the past, it was not a surprise that the Maximite and other video-generating solutions use direct memory access (DMA) to get the hardware, as opposed to programs, to read through memory and to write its contents to a destination, which in most cases seemed to be the memory address holding output data to be emitted via a data pin using the SPI mechanism.

I had also envisaged using DMA and was still fixated on using PMP to emit the different data bits to the output circuit producing the analogue signals for the display. Indeed, Microchip promotes the PMP and DMA combination as a way of doing “low-cost controllerless graphics solutions” involving LCD panels, so I felt that there surely couldn’t be much difference between that and getting an image on my monitor via a few resistors on the breadboard.

And so, a tour of different PIC32 features began, trying to understand the DMA documentation, the PMP documentation, all the while trying to get a grasp of what the VGA signal actually looks like, the timing constraints of the various synchronisation pulses, and battle various aspects of the MIPS architecture and the PIC32 implementation of it, constantly refining my own perceptions and understanding and learning perhaps too often that there may have been things I didn’t know quite enough about before trying them out!

Using VGA to Build a Picture

Before we really start to look at a VGA signal, let us first look at how a picture is generated by the signal on a screen:

VGA Picture Structure

The structure of a display image or picture produced by a VGA signal

The most important detail at this point is the central area of the diagram, filled with horizontal lines representing the colour information that builds up a picture on the display, with the actual limits of the screen being represented here by the bold rectangle outline. But it is also important to recognise that even though there are a number of visible “display lines” within which the colour information appears, the entire “frame” sent to the display actually contains yet more lines, even though they will not be used to produce an image.

Above and below – really before and after – the visible display lines are the vertical back and front porches whose lines are blank because they do not appear on the screen or are used to provide a border at the top and bottom of the screen. Such extra lines contribute to the total frame period and to the total number of lines dividing up the total frame period.

Figuring out how many lines a display will have seems to involve messing around with something called the “generalised timing formula”, and if you have an X server like Xorg installed on your system, you may even have a tool called “gtf” that will attempt to calculate numbers of lines and pixels based on desired screen resolutions and frame rates. Alternatively, you can look up some common sets of figures on sites providing such information.

What a VGA Signal Looks Like

Some sources show diagrams attempting to describe the VGA signal, but many of these diagrams are open to interpretation (in some cases, very much so). They perhaps show the signal for horizontal (display) lines, then other signals for the entire image, but they either do not attempt to combine them, or they instead combine these details ambiguously.

For instance, should the horizontal sync (synchronisation) pulse be produced when the vertical sync pulse is active or during the “blanking” period when no pixel information is being transmitted? This could be deduced from some diagrams but only if you share their authors’ unstated assumptions and do not consider other assertions about the signal structure. Other diagrams do explicitly show the horizontal sync active during vertical sync pulses, but this contradicts statements elsewhere such as “during the vertical sync period the horizontal sync must also be held low”, for instance.

After a lot of experimentation, I found that the following signal structure was compatible with the monitor I use with my computer:

VGA Signal Structure

The basic structure of a VGA signal, or at least a signal that my monitor can recognise

There are three principal components to the signal:

  • Colour information for the pixel or line data forms the image on the display and it is transferred within display lines during what I call the visible display period in every frame
  • The horizontal sync pulse tells the display when each horizontal display line ends, or at least the frequency of the lines being sent
  • The vertical sync pulse tells the display when each frame (or picture) ends, or at least the refresh rate of the picture

The voltage levels appear to be as follows:

  • Colour information should be at 0.7V (although some people seem to think that 1V is acceptable as “the specified peak voltage for a VGA signal”)
  • Sync pulses are supposed to be at “TTL” levels, which apparently can be from 0V to 0.5V for the low state and from 2.7V to 5V for the high state

Meanwhile, the polarity of the sync pulses is also worth noting. In the above diagram, they have negative polarity, meaning that an active pulse is at the low logic level. Some people claim that “modern VGA monitors don’t care about sync polarity”, but since it isn’t clear to me what determines the polarity, and since most descriptions and demonstrations of VGA signal generation seem to use negative polarity, I chose to go with the flow. As far as I can tell, the gtf tool always outputs the same polarity details, whereas certain resources provide signal characteristics with differing polarities.

It is possible, and arguably advisable, to start out trying to generate sync pulses and just grounding the colour outputs until your monitor (or other VGA-capable display) can be persuaded that it is receiving a picture at a certain refresh rate and resolution. Such confirmation can be obtained on a modern display by seeing a blank picture without any “no signal” or “input not supported” messages and by being able to activate the on-screen menu built into the device, in which an option is likely to exist to show the picture details.

How the sync and colour signals are actually produced will be explained later on. This section was merely intended to provide some background and gather some fairly useful details into one place.

Counting Lines and Generating Vertical Sync Pulses

The horizontal and vertical sync pulses are each driven at their own frequency. However, given that there are a fixed number of lines in every frame, it becomes apparent that the frequency of vertical sync pulse occurrences is related to the frequency of horizontal sync pulses, the latter occurring once per line, of course.

With, say, 622 lines forming a frame, the vertical sync will occur once for every 622 horizontal sync pulses, or at a rate that is 1/622 of the horizontal sync frequency or “line rate”. So, if we can find a way of generating the line rate, we can not only generate horizontal sync pulses, but we can also count cycles at this frequency, and every 622 cycles we can produce a vertical sync pulse.

But how do we calculate the line rate in the first place? First, we decide what our refresh rate should be. The “classic” rate for VGA output is 60Hz. Then, we decide how many lines there are in the display including those extra non-visible lines. We multiply the refresh rate by the number of lines to get the line rate:

60Hz * 622 = 37320Hz = 37.320kHz

On a microcontroller, the obvious way to obtain periodic events is to use a timer. Given a particular frequency at which the timer is updated, a quick calculation can be performed to discover how many times a timer needs to be incremented before we need to generate an event. So, let us say that we have a clock frequency of 24MHz, and a line rate of 37.320kHz, we calculate the number of timer increments required to produce the latter from the former:

24MHz / 37.320kHz = 24000000Hz / 37320Hz = 643

So, if we set up a timer that counts up to 642 and then upon incrementing again to 643 actually starts again at zero, with the timer sending a signal when this “wraparound” occurs, we can have a mechanism providing a suitable frequency and then make things happen at that frequency. And this includes counting cycles at this particular frequency, meaning that we can increment our own counter by 1 to keep track of display lines. Every 622 display lines, we can initiate a vertical sync pulse.

One aspect of vertical sync pulses that has not yet been mentioned is their duration. Various sources suggest that they should last for only two display lines, although the “gtf” tool specifies three lines instead. Our line-counting logic therefore needs to know that it should enable the vertical sync pulse by bringing it low at a particular starting line and then disable it by bringing it high again after two whole lines.

Generating Horizontal Sync Pulses

Horizontal sync pulses take place within each display line, have a specific duration, and they must start at the same time relative to the start of each line. Some video output demonstrations seem to use lots of precisely-timed instructions to achieve such things, but we want to use the peripherals of the microcontroller as much as possible to avoid wasting CPU time. Having considered various tricks involving specially formulated data that might be transferred from memory to act as a pulse, I was looking for examples of DMA usage when I found a mention of something called the Output Compare unit on the PIC32.

What the Output Compare (OC) units do is to take a timer as input and produce an output signal dependent on the current value of the timer relative to certain parameters. In clearer terms, you can indicate a timer value at which the OC unit will cause the output to go high, and you can indicate another timer value at which the OC unit will cause the output to go low. It doesn’t take much imagination to realise that this sounds almost perfect for generating the horizontal sync pulse:

  1. We take the timer previously set up which counts up to 643 and thus divides the display line period into units of 1/643.
  2. We identify where the pulse should be brought low and present that as the parameter for taking the output low.
  3. We identify where the pulse should be brought high and present that as the parameter for taking the output high.

Upon combining the timer and the OC unit, then configuring the output pin appropriately, we end up with a low pulse occurring at the line rate, but at a suitable offset from the start of each line.

VGA Display Line Structure

The structure of each visible display line in the VGA signal

In fact, the OC unit also proves useful in actually generating the vertical sync pulses, too. Although we have a timer that can tell us when it has wrapped around, we really need a mechanism to act upon this signal promptly, at least if we are to generate a clean signal. Unfortunately, handling an interrupt will introduce a delay between the timer wrapping around and the CPU being able to do something about it, and it is not inconceivable that this delay may vary depending on what the CPU has been doing.

So, what seems to be a reasonable solution to this problem is to count the lines and upon seeing that the vertical sync pulse should be initiated at the start of the next line, we can enable another OC unit configured to act as soon as the timer value is zero. Thus, upon wraparound, the OC unit will spring into action and bring the vertical sync output low immediately. Similarly, upon realising that the next line will see the sync pulse brought high again, we can reconfigure the OC unit to do so as soon as the timer value again wraps around to zero.

Inserting the Colour Information

At this point, we can test the basic structure of the signal and see if our monitor likes it. But none of this is very interesting without being able to generate a picture, and so we need a way of getting pixel information from the microcontroller’s memory to its outputs. We previously concluded that Direct Memory Access (DMA) was the way to go in reading the pixel data from what is usually known as a framebuffer, sending it to another place for output.

As previously noted, I thought that the Parallel Master Port (PMP) might be the right peripheral to use. It provides an output register, confusingly called the PMDIN (parallel master data in) register, that lives at a particular address and whose value is exposed on output pins. On the PIC32MX270, only the least significant eight bits of this register are employed in emitting data to the outside world, and so a DMA destination having a one-byte size, located at the address of PMDIN, is chosen.

The source data is the framebuffer, of course. For various retrocomputing reasons hinted at above, I had decided to generate a picture 160 pixels in width, 256 lines in height, and with each byte providing eight bits of colour depth (specifying how many distinct colours are encoded for each pixel). This requires 40 kilobytes and can therefore reside in the 64 kilobytes of RAM provided by the PIC32MX270. It was at this point that I learned a few things about the DMA mechanisms of the PIC32 that didn’t seem completely clear from the documentation.

Now, the documentation talks of “transactions”, “cells” and “blocks”, but I don’t think it describes them as clearly as it could do. Each “transaction” is just a transfer of a four-byte word. Each “cell transfer” is a collection of transactions that the DMA mechanism performs in a kind of batch, proceeding with these as quickly as it can until it either finishes the batch or is told to stop the transfer. Each “block transfer” is a collection of cell transfers. But what really matters is that if you want to transfer a certain amount of data and not have to keep telling the DMA mechanism to keep going, you need to choose a cell size that defines this amount. (When describing this, it is hard not to use the term “block” rather than “cell”, and I do wonder why they assigned these terms in this way because it seems counter-intuitive.)

You can perhaps use the following template to phrase your intentions:

I want to transfer <cell size> bytes at a time from a total of <block size> bytes, reading data starting from <source address>, having <source size>, and writing data starting at <destination address>, having <destination size>.

The total number of bytes to be transferred – the block size – is calculated from the source and destination sizes, with the larger chosen to be the block size. If we choose a destination size less than the source size, the transfers will not go beyond the area of memory defined by the specified destination address and the destination size. What actually happens to the “destination pointer” is not immediately obvious from the documentation, but for our purposes, where we will use a destination size of one byte, the DMA mechanism will just keep writing source bytes to the same destination address over and over again. (One might imagine the pointer starting again at the initial start address, or perhaps stopping at the end address instead.)

So, for our purposes, we define a “cell” as 160 bytes, being the amount of data in a single display line, and we only transfer one cell in a block. Thus, the DMA source is 160 bytes long, and even though the destination size is only a single byte, the DMA mechanism will transfer each of the source bytes into the destination. There is a rather unhelpful diagram in the documentation that perhaps tries to communicate too much at once, leading one to believe that the cell size is a factor in how the destination gets populated by source data, but the purpose of the cell size seems only to be to define how much data is transferred at once when a transfer is requested.

DMA Transfer Mechanism

The transfer of framebuffer data to PORTB using DMA cell transfers (noting that this hints at the eventual approach which uses PORTB and not PMDIN)

In the matter of requesting a transfer, we have already described the mechanism that will allow us to make this happen: when the timer signals the start of a new line, we can use the wraparound event to initiate a DMA transfer. It would appear that the transfer will happen as fast as both the source and the destination will allow, at least as far as I can tell, and so it is probably unlikely that the data will be sent to the destination too quickly. Once the transfer of a line’s pixel data is complete, we can do some things to set up the transfer for the next line, like changing the source data address to point to the next 160 bytes representing the next display line.

(We could actually set the block size to the length of the entire framebuffer – by setting the source size – and have the DMA mechanism automatically transfer each line in turn, updating its own address for the current line. However, I intend to support hardware scrolling, where the address of the first line of the screen can be adjusted so that the display starts part way through the framebuffer, reaches the end of the framebuffer part way down the screen, and then starts again at the beginning of the framebuffer in order to finish displaying the data at the bottom of the screen. The DMA mechanism doesn’t seem to support the necessary address wraparound required to manage this all by itself.)

Output Complications

Having assumed that the PMP peripheral would be an appropriate choice, I soon discovered some problems with the generated output. Although the data that I had stored in the RAM seemed to be emitted as pixels in appropriate colours, there were gaps between the pixels on the screen. Yet the documentation seemed to vaguely indicate that the PMDIN register was more or less continuously updated. That meant that the actual output signals were being driven low between each pixel, causing black-level gaps and ruining the result.

I wondered if anything could be done about this issue. PMP is really intended as some kind of memory interface, and it isn’t unreasonable for it to only maintain valid data for certain periods of time, modifying control signals to define this valid data period. That PMP can be used to drive LCD panels is merely a result of those panels themselves upholding this kind of interface. For those of you familiar with microcontrollers, the solution to my problem was probably obvious several paragraphs ago, but it needed me to reconsider my assumptions and requirements before I realised what I should have been doing all along.

Unlike SPI, which concerns itself with the bit-by-bit serial output of data, PMP concerns itself with the multiple-bits-at-once parallel output of data, and all I wanted to do was to present multiple bits to a memory location and have them translated to a collection of separate signals. But, of course, this is exactly how normal I/O (input/output) pins are provided on microcontrollers! They all seem to provide “PORT” registers whose bits correspond to output pins, and if you write a value to those registers, all the pins can be changed simultaneously. (This feature is obscured by platforms like Arduino where functions are offered to manipulate only a single pin at once.)

And so, I changed the DMA destination to be the PORTB register, which on the PIC32MX270 is the only PORT register with enough bits corresponding to I/O pins to be useful enough for this application. Even then, PORTB does not have a complete mapping from bits to pins: some pins that are available in other devices have been dedicated to specific functions on the PIC32MX270F256B and cannot be used for I/O. So, it turns out that we can only employ at most seven bits of our pixel data in generating signal data:

PORTB Pin Availability on the PIC32MX270F256B Pins … 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 … RPB15 RPB14 RPB13 RPB11 RPB10 RPB9 RPB8 RPB7 RPB5 RPB4 RPB3 RPB2 RPB1 RPB0

We could target the first byte of PORTB (bits 0 to 7) or the second byte (bits 8 to 15), but either way we will encounter an unmapped bit. So, instead of choosing a colour representation making use of eight bits, we have to make do with only seven.

Initially, not noticing that RPB6 was not available, I was using a “RRRGGGBB” or “332″ representation. But persuaded by others in a similar predicament, I decided to choose a representation where each colour channel gets two bits, and then a separate intensity bit is used to adjust the final intensity of the basic colour result. This also means that greyscale output is possible because it is possible to balance the channels.

The 2-bit-per-channel plus intensity colours

The colours employing two bits per channel plus one intensity bit, perhaps not shown completely accurately due to circuit inadequacies and the usual white balance issues when taking photographs

It is worth noting at this point that since we have left the 8-bit limitations of the PMP peripheral far behind us now, we could choose to populate two bytes of PORTB at once, aiming for sixteen bits per pixel but actually getting fourteen bits per pixel once the unmapped bits have been taken into account. However, this would double our framebuffer memory requirements for the same resolution, and we don’t have that much memory. There may be devices with more than sixteen bits mapped in the 32-bit PORTB register (or in one of the other PORT registers), but they had better have more memory to be useful for greater colour depths.

Back in Black

One other matter presented itself as a problem. It is all very well generating a colour signal for the pixels in the framebuffer, but what happens at the end of each DMA transfer once a line of pixels has been transmitted? For the portions of the display not providing any colour information, the channel signals should be held at zero, yet it is likely that the last pixel on any given line is not at the lowest possible (black) level. And so the DMA transfer will have left a stray value in PORTB that could then confuse the monitor, producing streaks of colour in the border areas of the display, making the monitor unsure about the black level in the signal, and also potentially confusing some monitors about the validity of the picture, too.

As with the horizontal sync pulses, we need a prompt way of switching off colour information as soon as the pixel data has been transferred. We cannot really use an Output Compare unit because that only affects the value of a single output pin, and although we could wire up some kind of blanking in our external circuit, it is simpler to look for a quick solution within the capabilities of the microcontroller. Fortunately, such a quick solution exists: we can “chain” another DMA channel to the one providing the pixel data, thereby having this new channel perform a transfer as soon as the pixel data has been sent in its entirety. This new channel has one simple purpose: to transfer a single byte of black pixel data. By doing this, the monitor will see black in any borders and beyond the visible regions of the display.

Wiring Up

Of course, the microcontroller still has to be connected to the monitor somehow. First of all, we need a way of accessing the pins of a VGA socket or cable. One reasonable approach is to obtain something that acts as a socket and that breaks out the different signals from a cable, connecting the microcontroller to these broken-out signals.

Wanting to get something for this task quickly and relatively conveniently, I found a product at a local retailer that provides a “male” VGA connector and screw-adjustable terminals to break out the different pins. But since the VGA cable also has a male connector, I also needed to get a “gender changer” for VGA that acts as a “female” connector in both directions, thus accommodating the VGA cable and the male breakout board connector.

Wiring up to the broken-out VGA connector pins is mostly a matter of following diagrams and the pin numbering scheme, illustrated well enough in various resources (albeit with colour signal transposition errors in some resources). Pins 1, 2 and 3 need some special consideration for the red, green and blue signals, and we will look at them in a moment. However, pins 13 and 14 are the horizontal and vertical sync pins, respectively, and these can be connected directly to the PIC32 output pins in this case, since the 3.3V output from the microcontroller is supposedly compatible with the “TTL” levels. Pins 5 through 10 can be connected to ground.

We have seen mentions of colour signals with magnitudes of up to 0.7V, but no explicit mention of how they are formed has been presented in this article. Fortunately, everyone is willing to show how they converted their digital signals to an analogue output, with most of them electing to use a resistor network to combine each output pin within a channel to produce a hopefully suitable output voltage.

Here, with two bits per channel, I take the most significant bit for a channel and send it through a 470ohm resistor. Meanwhile, the least significant bit for the channel is sent through a 1000ohm resistor. Thus, the former contributes more to the magnitude of the signal than the latter. If we were only dealing with channel information, this would be as much as we need to do, but here we also employ an intensity bit whose job it is to boost the channels by a small amount, making sure not to allow the channels to pollute each other via this intensity sub-circuit. Here, I feed the intensity output through a 2200ohm resistor and then to each of the channel outputs via signal diodes.

VGA Output Circuit

The circuit showing connections relevant to VGA output (generic connections are not shown)

The Final Picture

I could probably go on and cover other aspects of the solution, but the fundamental aspects are probably dealt with sufficiently above to help others reproduce this experiment themselves. Populating memory with usable image data, at least in this solution, involves copying data to RAM, and I did experience problems with accessing RAM that are probably related to CPU initialisation (as covered in my previous article) and to synchronising the memory contents with what the CPU has written via its cache.

As for the actual picture data, the RGB-plus-intensity representation is not likely to be the format of most images these days. So, to prepare data for output, some image processing is needed. A while ago, I made a program to perform palette optimisation and dithering on images for the Acorn Electron, and I felt that it was going to be easier to adapt the dithering code than it was to figure out the necessary techniques required for software like ImageMagick or the Python Imaging Library. The pixel data is then converted to assembly language data definition statements and incorporated into my PIC32 program.

VGA output from a PIC32 microcontroller

VGA output from a PIC32 microcontroller, featuring a picture showing some Oslo architecture, with the PIC32MX270 being powered (and programmed) by the Arduino Duemilanove, and with the breadboards holding the necessary resistors and diodes to supply the VGA breakout and, beyond that, the cable to the monitor

To demonstrate control over the visible region, I deliberately adjusted the display frequencies so that the monitor would consider the signal to be carrying an image 800 pixels by 600 pixels at a refresh rate of 60Hz. Since my framebuffer is only 256 lines high, I double the lines to produce 512 lines for the display. It would seem that choosing a line rate to try and produce 512 lines has the monitor trying to show something compatible with the traditional 640×480 resolution and thus lines are lost off the screen. I suppose I could settle for 480 lines or aim for 300 lines instead, but I actually don’t mind having a border around the picture.

The on-screen menu showing the monitor's interpretation of the signal

The on-screen menu showing the monitor's interpretation of the signal

It is worth noting that I haven’t really mentioned a “pixel clock” or “dot clock” so far. As far as the display receiving the VGA signal is concerned, there is no pixel clock in that signal. And as far as we are concerned, the pixel clock is only important when deciding how quickly we can get our data into the signal, not in actually generating the signal. We can generate new colour values as slowly (or as quickly) as we might like, and the result will be wider (or narrower) pixels, but it shouldn’t make the actual signal invalid in any way.

Of course, it is important to consider how quickly we can generate pixels. Previously, I mentioned a 24MHz clock being used within the PIC32, and it is this clock that is used to drive peripherals and this clock’s frequency that will limit the transfer speed. As noted elsewhere, a pixel clock frequency of 25MHz is used to support the traditional VGA resolution of 640×480 at 60Hz. With the possibilities of running the “peripheral clock” in the PIC32MX270 considerably faster than this, it becomes a matter of experimentation as to how many pixels can be supported horizontally.

Some Oslo street art being displayed by the PIC32

Some Oslo street art being displayed by the PIC32

For my own purposes, I have at least reached the initial goal of generating a stable and usable video signal. Further work is likely to involve attempting to write routines to modify the framebuffer, maybe support things like scrolling and sprites, and even consider interfacing with other devices.

Naturally, this project is available as Free Software from its own repository. Maybe it will inspire or encourage you to pursue something similar, knowing that you absolutely do not need to be any kind of “expert” to stubbornly persist and to eventually get results!

Free Software – Frank Karlitschek_: The development of Global Scale

The architecture of Nextcloud is a classic Web Application architecture. I picked this architecture 7.5 years ago because it is very well known and is proven to be scaled relatively easily. This usually works with off the shelf technologies like http load balancers, clusters of Linux webservers and clustered databases.

But for many years users and customers asked for ways to distribute a single instance over several datacenters. A lot of users and customers run organizations that are not in once office or sometimes not even in once country or on one continent. So how can the service run distributed over different hosting centers on different continent?

Until now there was no good answer for this requirement.

Over the years I talked with users who experimented with different approaches. Unfortunately, they all didn’t work.

If you talk to storage people how to solve this challenge they say: No problem. Just use a distributed storage system like Gluster, Ceph, Hadoop or other.

If you talk to database people and how to do this they say: No problem! Just use one of the cluster and replication systems for Oracle, MySQL, MariaDB and others.

The challenge is how this all works together. If a user changes a file in Nextcloud then it is necessary that the file is changed, potential encryption keys are updated, log files are written, database tables are changed, email notification are send, external workflow scripts are executed and a lot of other things happen. All this operations that are triggered by a single file change have to happen in an atomic way. It happens completely or not at all. Using database replication and storage replication independently will lead to a broken state and data loss. Additional problems are that you need the full bandwidth and storage in all data centers. So there is a lot of overhead.

The Global Scale architecture, that we designed, is currently the only solution, as far as I know, which solves this challenge.

Additional benefits are that the storage and database and overall operational costs decreases because simpler, commodity and more standard components can be used.

Another benefit is that the locality of data can be controlled. So if it is a legal requirement that certain files never leave a certain jurisdiction then this can be guaranteed with the right GS Balancer and File Access Control settings.

So far I only talk about the benefit that GS breaks a service down into several data centers. This is only half of the truth. Nextcloud Global Scale can be used in an even more radical way. You could stop using clustered Nextcloud instances in general. Killing all central storage boxes and databases and move completely to commodity hardware. Using only local storage and local databases and local caching. This changes the world completely and makes big storages, SAN, NFS and object store boxes completely obsolete.

The Global Scale architecture idea and implementation was developed over the last year with feedback and ideas from many different people. Storage experts, Database vendors, Linux distribution people, container and orchestration experts for easy automatic deployment and several big customers and users of Nextcloud.

The main inspiration came out of long discussions with the people at DeiC who is running the National Research and Education Network of Denmark. DeiC was doing successful experiments with a similar architecture for a while already.

At Nextcloud we are committed to develop everything we do as open source. This includes this feature. Also, if you want to contribute to this architecture then no Contributer License Agreement is needed and you don’t need to transfer any rights to the Nextcloud company.

More information about Nextcloud Global Scale can be found here: https://nextcloud.com/globalscale

May 20 2017

vanitasvitae's blog » englisch: GSoC: Second, third week of community bonding

Hi all!

This is my report for the second, as well as the first half of the third week of GSoC community bonding, which I spent again working on finalizing my OMEMO code.

I dug deeper into writing test cases, mainly integration tests and I found quite a lot of small, undetectable bugs this way (Yay!). This strengthens my plan to work test driven during my GSoC project. The OMEMO code I currently work on was started as part of my bachelor thesis about 4 to 5 months ago and at this time, I was more concerned about having working code in the end, so I wrote no tests at all. Deploying test cases AFTER the code is already written is not only a tideous task, but its also often very difficult (because the code is not structured properly). So I learned my lesson the hard way :D

During testing I also found another bug in an XMPP server software, which prevents Smack from creating accounts on the server on the fly. Unfortunatelly this bug will not get fixed anymore for the version I use (installed from debian testing repository, which I thought was *reasonable* new), which keeps me from doing proper testing the way its meant to be done. I don’t have the time to compile the server software myselves. Instead, I work around this issue by creating the accounts manually everytime I run the test suite using a small bashscript.

I also had to deal with a really strange bug with file writing and reading. smack-omemo has a set of 4 integration tests, which all write data into a temporary directory. After each test, the directory is deleted to prevent tests influencing eachother. The issue was, that only the first test could read/write to the test directory. All subsequent tests failed for some reason. It took me a long time to notice, that there were two folders created (one in the working directory, another one in the subdirectory of the integration test framework). I am still not really sure what happened. The first folder was logged in all debug output, while files were written (by the first test) to the second filder. I guess it was caused by the temp directory being specified using a relative path, which messed up the tests, which were instanciated by the test framework using reflection. But I’m really not sure about this. Specifying the directory using an absolute path fixed the issue in the end.

Last but not least, me and Flow worked out some more details about my GSoC project (Implementing (encrypted) Jingle file transfer for Smack). The module will most likely be based upon java.nio to be scalable in the future. Flow also emphasized that the API should be as easy to use as possible, but at the same time powerful and extensible, which is a nice challenge (and probably a common one within the XMPP community). My initial plan was to create a XEP for OMEMO encrypted Jingle file transfer. We decided, that it would be of more value, to specify the XEP in a way, which allows arbitrary encryption techniques instead of being OMEMO exclusive.

Currently there is a little bit of tension in the community regarding the OMEMO specification. I really hope (and believe) there is a solution which is suitable of making everybody happy and I’m looking forward to participate in an open discussion :)

Happy hacking!

Daniel's FSFE blog: Benchmarking microSD cards and more

It’s worth re-visiting this blog-post as I am going to do more benchmarks and add more results over time.


If you ever tried using a flash card (such a SD, microSD or USB drive) for your root or home filesystem on a small computing device or smartphone, you probably have noticed that flash cards are in most cases a lot slower than integreted eMMC flash. Since most filesystems use 4k blocks, the random write/read performance using 4k blocks is what matters most in such scenarios. And while flash cards don’t come close to internal flash in these disciplines, there are significant differences between the models.

Jeff Geerling [1,2] has already benchmarked the performance of various microSD cards on different models of the “blobby” Raspberry Pi. I had a number of different microSD cards at hand and I tried to replicate his results on my sample. In addition, I extended the comparison to regular SD cards and USB drives.


All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

Environment and tools

I encourage every reader to replicate my benchmarks and share their results. Therefore, I first want to describe how the measures were taken.

General equipment

For the benchmarks, I used the following hardware on my workstation (listing only relevant parts):

  • Logilink CR0015 USB 2.0 card reader [4]
  • Logilink CR0034A USB 3.0 card reader [5]
  • Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
  • Debian 9 “Stretch”

iozone Benchmarks

For the first tests, Just like Jeff, I used the open source (but non-free) tool “iozone” [3] in the current stable version that is available on Debian Stretch (3.429).

I disabled caches and benchmarked on raw devices to avoid measuring filesystem overhead. Therefore, I used the following call to iozone to run the benchmarks (/dev/sde is my sdcard):

$ iozone -e -I -a -s 100M -r 4k -r 16M -i 0 -i 1 -i 2 -f /dev/sdX

(replacing sdX by the actual device name)

ddrescue benchmarks

I used GNU ddrescue (1.21) in synchronous mode to overwrite a device with zeroes and measured the performance there. For this purpose, I used the following call to do a write benchmark:

$ ddrescue -D --force /dev/zero /dev/sdX

(replacing sdX by the actual device name)

Similarly, to measure the read performance I used (I had more than 100GB RAM left for tmpfs):

$ ddrescue -D --force /dev/sdX /tmp/foo

(replacing sdX by the actual device name)



The following table provides the results I obtained with the USB 2.0 card reader (rounded to 10kb/s):

Manufacturer Make / model Type Rating Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s) Adata ? mSD C4 32 GB 9.58 2.97 2.62 0.64 Samsung Evo mSD UHS-1 32 GB 17.80 10.13 4.45 1.32 Samsung Evo+ mSD UHS-1 32 GB 18.10 14.27 5.28 2.86 Samsung Evo+ (2017) mSD UHS-3 64 GB 16,98 8,03 7,99 2,99 Sandisk Extreme mSD UHS-3 64 GB 18.44 11.22 5.20 2.36 Sandisk Ultra mSD C10 64 GB 18.16 8.69 3.73 0.80 Toshiba Exceria mSD UHS-3 64 GB 16.10 6.60 3.82 0.09 Kingston ? mSD C4 8 GB 14.69 3.71 3.97 0.18

The next table provides the results I obtained with the USB 3.0 card reader (also rounded to 10kb/s):

Manufacturer Make / model Type Rating Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s) Samsung Evo+ (2017) mSD UHS-3 64 GB 42,47 40,10 8,48 2,29 Samsung Evo+ mSD UHS-1 32 GB 42,53 34,48 8,01 3,97 Sandisk Ultra mSD C10 64 GB 18,21 19,19 5,35 1,06

And here are the USB drives:

Manufacturer Make / model Type Remarks Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s) Sandisk Cruzer USB 2.0 - 4 GB 22,89 4,05 6,86 0,44 Kingston Data Traveller G3 USB 2.0 - 8 GB 21,52 4,96 6,96 0,08 Super Talent Pico C USB 2.0 gold 8 GB 36,72 13,77 DNF DNF


Using the USB 3.0 card reader, I obtained the following results:

Manufacturer Make / model Type Rating Capacity Avg read (MB/s) Avg write (MB/s) Sandisk Evo+ mSD C4 32 GB 41,95 21,54

Various USB drives:

Manufacturer Make / model Type Remarks Capacity Avg read (MB/s) Avg write (MB/s) Sandisk Cruzer USB 2.0 - 4 GB 36,37 2,64 Super Talent Pico C USB 2.0 gold 8 GB 36,29 19,80

Verdict and future work

I can confirm Jeff’s results about microSD cards and would also recommend the Evo+ (both the old and the 2017 model) which have the best 4K random write performance of the sample. On the other hand, I am very disappointed about the Toshiba Exceria card. Actually running a device on this card with a very sluggish performance was the reason why I took this benchmark initiative. And indeed, after switching to the Evo+, the device feels much snappier now.

Of course, it would be worthwhile to add more cards and drives to this benchmark. Also, using fio instead of the non-free iozone might be interesting. Furthermore, doing the benchmarks internally on the device or using a other USB 3.0 card readers might be also interesting.


[1] http://www.pidramble.com/wiki/benchmarks/microsd-cards
[2] http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card
[3] http://www.iozone.org/
[4] http://www.logilink.eu/Products_LogiLink/Notebook-Computer_Accessories/Card_Reader/Cardreader_USB_20_Stick_external_for_SD-MMC_CR0015.htm
[5] http://www.logilink.org/Produkte_LogiLink/Notebook-Computerzubehoer/Kartenleser/Cardreader_USB_30_SD-SD-HC-Micro_SD-Micro_SD-HC_CR0034A.htm?seticlanguage=en

May 19 2017

Paul Boddie's Free Software-related blog » English: Evaluating PIC32 for Hardware Experiments

Some time ago I became aware of the PIC32 microcontroller platform, perhaps while following various hardware projects, pursuing hardware-related interests, and looking for pertinent documentation. Although I had heard of PIC microcontrollers before, my impression of them was that they were mostly an alternative to other low-end computing products like the Atmel AVR series, but with a development ecosystem arguably more reliant on its vendor than the Atmel products for which tools like avr-gcc and avrdude exist as Free Software and have gone on to see extensive use, perhaps most famously in connection with the Arduino ecosystem.

What made PIC32 stand out when I first encountered it, however, was that it uses the MIPS32 instruction set architecture instead of its own specialised architecture. Consequently, instead of being reliant on the silicon vendor and random third-party tool providers for proprietary tools, the possibility of using widely-used Free Software tools exists. Moreover, with a degree of familiarity with MIPS assembly language, thanks to the Ben NanoNote, I felt that there might be an opportunity to apply some of my elementary skills to another MIPS-based system and gain some useful experience.

Some basic investigation occurred before I made any attempt to acquire hardware. As anyone having to pay attention to the details of hardware can surely attest, it isn’t much fun to obtain something only to find that some necessary tool required for the successful use of that hardware happens to be proprietary, only works on proprietary platforms, or is generally a nuisance in some way that makes you regret the purchase. Looking around at various resources such as the Pinguino Web site gave me some confidence that there were people out there using PIC32 devices with Free Software. (Of course, the eventual development scenario proved to be different from that envisaged in these initial investigations, but previous experience has taught me to expect that things may not go as originally foreseen.)

Some Discoveries

So that was perhaps more than enough of an introduction, given that I really want to focus on some of my discoveries in using my acquired PIC32 devices, hoping that writing them up will help others to use Free Software with this platform. So, below, I will present a few discoveries that may well, for all I know, be “obvious” to people embedded in the PIC universe since it began, or that might be “superfluous” to those happy that Microchip’s development tools can obscure the operation of the hardware to offer a simpler “experience”.

I should mention at this point that I chose to acquire PDIP-profile products for use with a solderless breadboard. This is the level of sophistication at which the Arduino products mostly operate and it allows convenient prototyping involving discrete components and other electronic devices. The evidence from the chipKIT and Pinguino sites suggested that it would be possible to set up a device on a breadboard, wire it up to a power supply with a few supporting components, and then be able to program it. (It turned out that programming involved another approach than indicated by that latter reference, however.)

The 28-pin product I elected to buy was the PIC32MX270F256B-50/SP. I also bought some capacitors that were recommended for wiring up the device. In the picture below, you can just about see the capacitor connecting two of the pins, and there is also a resistor pulling up one of the pins. I recommend obtaining a selection of resistors of different values so as to be able to wire up various circuits as you encounter them. Note that the picture does not show a definitive wiring guide: please refer to the product documentation or to appropriate project documentation for such things.

PIC32 device on a mini-breadboard

PIC32 device on a mini-breadboard connected to an Arduino Duemilanove for power and programming

Programming the Device

Despite the apparent suitability of a program called pic32prog, recommended by the “cheap DIY programmer” guide, I initially found success elsewhere. I suppose it didn’t help that the circuit diagram was rather hard to follow for me, as someone who isn’t really familiar with certain electrical constructs that have been mixed in, arguably without enough explanation.

Initial Recommendation: ArduPIC32

Instead, I looked for a solution that used an Arduino product (not something procured from ephemeral Chinese “auction site” vendors) and found ArduPIC32 living a quiet retirement in the Google Code Archive. Bypassing tricky voltage level conversion and connecting an Arduino Duemilanove with the PIC32 device on its 5V-tolerant JTAG-capable pins, ArduPIC32 mostly seemed to work, although I did alter it slightly to work the way I wanted to and to alleviate the usual oddness with Arduino serial-over-USB connections.

However, I didn’t continue to use ArduPIC. One reason is that programming using the JTAG interface is slow, but a much more significant reason is that the use of JTAG means that the pins on the PIC32 associated with JTAG cannot be used for other things. This is either something that “everybody” knows or just something that Microchip doesn’t feel is important enough to mention in convenient places in the product documentation. More on this topic below!

Final Recommendation: Pickle (and Nanu Nanu)

So, I did try and use pic32prog and the suggested circuit, but had no success. Then, searching around, I happened to find some very useful resources indeed: Pickle is a GPL-licensed tool for uploading data to different PIC devices including PIC32 devices; Nanu Nanu is a GPL-licensed program that runs on AVR devices and programs PIC32 devices using the ICSP interface. Compiling the latter for the Arduino and uploading it in the usual way (actually done by the Makefile), it can then run on the Arduino and be controlled by the Pickle tool.

Admittedly, I did have some problems with the programming circuit, most likely self-inflicted, but the developer of these tools was very responsive and, as I know myself from being in his position in other situations, provided the necessary encouragement that was perhaps most sorely lacking to get the PIC32 device talking. I used the “sink” or “open drain” circuit so that the Arduino would be driving the PIC32 device using a suitable voltage and not the Arduino’s native 5V. Conveniently, this is the default configuration for Nanu Nanu.

PIC32 on breadboard with Arduino programming circuit

PIC32 on breadboard with Arduino programming circuit (and some LEDs for diagnostic purposes)

I should point out that the Pinguino initiative promotes USB programming similar to that employed by the Arduino series of boards. Even though that should make programming very easy, it is still necessary to program the USB bootloader on the PIC32 device using another method in the first place. And for my experiments involving an integrated circuit on a breadboard, setting up a USB-based programming circuit is a distraction that would have complicated the familiarisation process and would have mostly duplicated the functionality that the Arduino can already provide, even though this two-stage programming circuit may seem a little contrived.

Compiling Code for the Device

This is perhaps the easiest problem to solve, strongly motivating my choice of PIC32 in the first place…

Recommendation: GNU Toolchain

PIC32 uses the MIPS32 instruction set. Since MIPS has been around for a very long time, and since the architecture was prominent in workstations, servers and even games consoles in the late 1980s and 1990s, remaining in widespread use in more constrained products such as routers as this century has progressed, the GNU toolchain (GCC, binutils) has had a long time to comfortably support MIPS. Although the computer you are using is not particularly likely to be MIPS-based, cross-compiling versions of these tools can be built to run on, say, x86 or x86-64 while generating MIPS32 executable programs.

And fortunately, Debian GNU/Linux provides the mipsel-linux-gnu variant of the toolchain packages (at least in the unstable version of Debian) that makes the task of building software simply a matter of changing the definitions for the compiler, linker and other tools in one’s Makefile to use these variants instead of the unprefixed “host” gcc, ld, and so on. You can therefore keep using the high-quality Free Software tools you already know. The binutils-mipsel-linux-gnu package provides an assembler, if you just want to practise your MIPS assembly language, as well as a linker and tools for creating binaries. Meanwhile, the gcc-mipsel-linux-gnu package provides a C compiler.

Perhaps the only drawback of using the GNU tools is that people using the proprietary tools supplied by Microchip and partners will post example code that uses special notation interpreted in a certain way by those products. Fortunately, with some awareness that this is going on, we can still support the necessary functionality in our own way, as described below.

Configuring the Device

With the PIC32, and presumably with the other PIC products, there is a distinct activity of configuring the device when programming it with program code and data. This isn’t so obvious until one reads the datasheets, tries to find a way of changing some behaviour of the device, and then stumbles across the DEVCFG registers. These registers cannot be set in a running program: instead, they are “programmed” before the code is run.

You might wonder what the distinction is between “programming” the device to take the code you have written and “programming” the configuration registers, and there isn’t much difference conceptually. All that matters is that you have your program written into one part of the device’s memory and you also ask for certain data values to be written to the configuration registers. How this is done in the Microchip universe is with something like this:

#pragma config SOMETHING = SOMEVALUE;

With a basic GNU tool configuration, we need to find the equivalent operation and express it in a different way (at least as far as I know, unless someone has implemented an option to support this kind of notation in the GNU tools). The mechanism for achieving this is related to the linker script and is described in a section of this article presented below. For now, we will concentrate on what configuration settings we need to change.

Recommendation: Disable JTAG

As briefly mentioned above, one thing that “everybody” knows, at least if they are using Microchip’s own tools and are busy copying code from Microchip’s examples, is that the JTAG functionality takes over various pins and won’t let you use them, even if you switch on a “peripheral” in the microcontroller that needs to use those pins. Maybe I am naive about how intrusive JTAG should or should not be, but the lesson from this matter is just to configure the device to not have JTAG features enabled and to use the ICSP programming method recommended above instead.

Recommendation: Disable Watchdog Timer

Again, although I am aware of the general concept of a watchdog timer – something that resets a device if it thinks that the device has hung, crashed, or experienced something particularly bad – I did not expect something like this to necessarily be configured by default. In case it is, and one does see lots of code assuming so, then it should be disabled as well. Otherwise, I can imagine that you might experience spontaneous resets for no obvious reason.

Recommendation: Configure the Oscillator and Clocks

If you need to change the oscillator frequency or the origin of the oscillator used by the PIC32, it is perhaps best to do this in the configuration registers rather than try and mess around with this while the device is running. Indeed, some configuration is probably unavoidable even if there is a need to, say, switch between oscillators at run-time. Meanwhile, the relationship between the system clock (used by the processor to execute instructions) and the peripheral clock (used to interact with other devices and to run things like timers) is defined in the configuration registers.

Linker Scripts

So, to undertake the matter of configuration, a way is needed to express the setting of configuration register values in a general way. For this, we need to take a closer look at linker scripts. If you routinely compile and link software, you will be using linker scripts without realising it, because such scripts are telling the tools things about where parts of the program should be stored, what kinds of addresses they should be using, and so on. Here, we need to take more of an interest in such matters.

Recommendation: Expand a Simple Script

Writing linker scripts does not seem like much fun. The syntax is awkward to read and to produce as a human being, and knowledge about tool output is assumed. However, it is possible to start with a simple script that works for someone else in a similar situation and to modify it conservatively in order to achieve what you need. I started out with one that just defined the memory regions and a few sections. To avoid reproducing all the details here, I will just show what the memory regions for a configuration register look like:

  config2                    : ORIGIN = 0xBFC00BF4, LENGTH = 0x4
  physical_config2           : ORIGIN = 0x3FC00BF4, LENGTH = 0x4

This will be written in the MEMORY construct in the script. What they tell us is that config2 is a region of four bytes starting at virtual address 0xBFC00BF4, which is the location of the DEVCFG2 register as specified in the PIC32MX270 documentation. However, this register actually has a physical address of 0x3FC00BF4. The difference between virtual addresses and physical addresses is perhaps easiest to summarise by saying that CPU instructions would use the virtual address when referencing the register, whereas the actual memory of the device employs the physical address to refer to this four-byte region.

Meanwhile, in the SECTIONS construct, there needs to be something like this:

  .devcfg2 : {
        } > config2 AT > physical_config2

Now you might understand my remark about the syntax! Nevertheless, what these things do is to tell the tools to put things from a section called .devcfg2 in the physical_config2 memory region and, if there were to be any address references in the data (which there isn’t in this case), then they would use the addresses in the config2 region.

Recommendation: Define Configuration Sections and Values in the Code

Since I have been using assembly language, here is what I do in my actual program source file. Having looked at the documentation and figured out which configuration register I need to change, I introduce a section in the code that defines the register value. For DEVCFG2, it looks like this:

.section .devcfg2, "a"
.word 0xfff9fffb        /* DEVCFG2<18:16> = FPLLODIV<2:0> = 001;
                        DEVCFG2<6:4> = FPLLMUL<2:0> = 111;
                        DEVCFG2<2:0> = FPLLIDIV<2:0> = 011 */

Here, I fully acknowledge that this might not be the most optimal approach, but when you’re learning about all these things at once, you make progress as best you can. In any case, what this does is to tell the assembler to include the .devcfg2 section and to populate it with the specified “word”, which is four bytes on the 32-bit PIC32 platform. This word contains the value of the register which has been expressed in hexadecimal with the most significant digit first.

Returning to our checklist of configuration tasks, what we now need to do is to formulate each one in terms of configuration register values, introduce the necessary sections and values, and adjust the values to contain the appropriate bit patterns. The above example shows how the DEVCFG2 bits are adjusted and set in the value of the register. Here is a short amplification of the operation:

DEVCFG2 Bits 31…28 27…24 23…20 19…16 15…12 11…8 7…4 3…0 1111 1111 1111 1001
FPLLODIV<2:0> 1111 1111 1111
FPLLMUL<2:0> 1011
FPLLIDIV<2:0> f f f 9 f f f b

Here, the underlined bits are those of interest and have been changed to the desired values. It turns out that we can set the other bits as 1 for the functionality we want (or don’t want) in this case.

By the way, the JTAG functionality is disabled in the DEVCFG0 register (JTAGEN, bit 2, on this device). The watchdog timer is disabled in DEVCFG1 (FWDTEN, bit 23, on this device).

Recommendation: Define Regions for Exceptions and Interrupts

The MIPS architecture has the processor jump to certain memory locations when bad things happen (exceptions) or when other things happen (interrupts). We will cover the details of this below, but while we are setting up the places in memory where things will reside, we might as well define where the code to handle exceptions and interrupts will be living:

  .flash : { *(.flash*) } > kseg0_program_mem AT > physical_program_mem

This will be written in the SECTIONS construct. It relies on a memory region being defined, which would appear in the MEMORY construct as follows:

  kseg0_program_mem    (rx)  : ORIGIN = 0x9D000000, LENGTH = 0x40000
  physical_program_mem (rx)  : ORIGIN = 0x1D000000, LENGTH = 0x40000

These definitions allow the .flash section to be placed at 0x9D00000 but actually be written to memory at 0x1D00000.

Initialising the Device

On the various systems I have used in the past, even when working in assembly language I never had to deal with the earliest stages of the CPU’s activity. However, on the MIPS systems I have used in more recent times, I have been confronted with the matter of installing code to handle system initialisation, and this does require some knowledge of what MIPS processors would like to do when something goes wrong or if some interrupt arrives and needs to be processed.

The convention with the PIC32 seems to be that programs are installed within the MIPS KSEG0 region (one of the four principal regions) of memory, specifically at address 0x9FC00000, and so in the linker script we have MEMORY definitions like this:

  kseg0_boot_mem       (rx)  : ORIGIN = 0x9FC00000, LENGTH = 0xBF0
  physical_boot_mem    (rx)  : ORIGIN = 0x1FC00000, LENGTH = 0xBF0

As you can see, this region is far shorter than the 512MB of the KSEG0 region in its entirety. Indeed, 0xBF0 is only 3056 bytes! So, we need to put more substantial amounts of code elsewhere, some of which will be tasked with handling things when they go wrong.

Recommendation: Define Exception and Interrupt Handlers

As we have seen, the matter of defining routines to handle errors and interrupt conditions falls on the unlucky Free Software developer in this case. When something goes wrong, like the CPU not liking an instruction it has just seen, it will jump to a predefined location and try and execute code to recover. By default, with the PIC32, this location will be at address 0×80000000 which is the start of RAM, but if the RAM has not been configured then the CPU will not achieve very much trying to load instructions from that address.

Now, it can be tempting to set the “exception base”, as it is known, to be the place where our “boot” code is installed (at 0x9FC00000). So if something bad does happen, our code will start running again “from the top”, a bit like what used to happen if you ever wrote BASIC code that said…


Clearly, this isn’t particularly desirable because it can mask problems. Indeed, I found that because I had not observed a step in the little dance required to change interrupt locations, my program would be happily restarting itself over and over again upon receiving interrupts. This is where the work done in the linker script starts to pay off: we can move the exception handler, this being the code residing at the exception base, to another region of memory and tell the CPU to jump to that address instead. We should therefore never have unscheduled restarts occurring once this is done.

Again, I have been working in assembly language, so I position my exception handling code using a directive like this:

.section .flash, "a"

    /* Exception handling code goes here. */

Note that .flash is what we mentioned in the linker script. After this directive, the exception handler is defined so that the CPU has something to jump to. But exceptions are just one kind of condition that may occur, and we also need to handle interrupts. Although we might be able to handle both things together, you can instead position an interrupt handler after the exception handler at a well-defined offset, and the CPU can be told to use that when it receives an interrupt. Here is what I do:

.org 0x200

    /* Interrupt handling code goes here. *

The .org directive tells the assembler that the interrupt handler will reside at an offset of 0×200 from the start of the .flash section. This number isn’t something I have made up: it is defined by the MIPS architecture and will be observed by the CPU when suitably configured.

So that leaves the configuration. Although one sees a lot of advocacy for the “multi-vector” interrupt handling support of the PIC32, possibly because the Microchip example code seems to use it, I wanted to stick with the more widely available “single-vector” support which is what I have effectively described above: you have one place the CPU jumps to and it is then left to the code to identify the interrupt condition – what it is that happened, exactly – and then handle the condition appropriately. (Multi-vector handling has the CPU identify the kind of condition and then choose a condition-specific location to jump to all by itself.)

The following steps are required for this to work properly:

  1. Make sure that the CP0 (co-processor 0) STATUS register has the BEV bit set. Otherwise things will fail seemingly inexplicably.
  2. Set the CP0 EBASE (exception base) register to use the exception handler address. This changes the target of the jump that occurs when an exception or interrupt occurs.
  3. Set the CP0 INTCTL (interrupt control) register to use a non-zero vector spacing, even though this setting probably has no relevance to the single-vector mode.
  4. Make sure that the CP0 CAUSE (exception cause) register has the IV bit set. This tells the CPU that the interrupt handler is at that magic 0×200 offset from the exception handler, meaning that interrupts should be dispatched there instead of to the exception handler.
  5. Now make sure that the CP0 STATUS register has the BEV bit cleared, so that the CPU will now use the new handlers.

To enable exceptions and interrupts, the IE bit in the CP0 STATUS register must be set, but there are also other things that must be done for them to actually be delivered.

Recommendation: Handle the Initial Error Condition

As I found out with the Ben NanoNote, a MIPS device will happily run in its initial state, but it turns out that this state is some kind of “error” state that prevents exceptions and interrupts from being delivered, even though interrupts may have been enabled. This does make some kind of sense: if a program is in the process of setting up interrupt handlers, it doesn’t really want an interrupt to occur before that work is done.

So, the MIPS architecture defines some flags in the CP0 STATUS register to override normal behaviour as some kind of “fail-safe” controls. The ERL (error level) bit is set when the CPU starts up, preventing errors (or probably most errors, at least) as well as interrupts from interrupting the execution of the installed code. The EXL (exception level) bit may also be set, preventing exceptions and interrupts from occurring even when ERL is clear. Thus, both of these bits must be cleared for interrupts to be enabled and delivered, and what happens next rather depends on how successful we were in setting up those handlers!

Recommendation: Set up the Global Offset Table

Something that I first noticed when looking at code for the Ben NanoNote was the initialisation of a register to refer to something called the global offset table (or GOT). For anyone already familiar with MIPS assembly language or the way code is compiled for the architecture, programs follow a convention where they refer to objects via a table containing the addresses of those objects. For example:

Global Offset Table Offset from Start Member +0 0 +4 address of _start +8 address of my_routine +12 address of other_routine … …

But in order to refer to the table, something has to have the address of the table. This is where the $gp register comes in: it holds that address and lets the code access members of the table relative to the address.

What seems to work for setting up the $gp register is this:

        lui $gp, %hi(_GLOBAL_OFFSET_TABLE_)
        ori $gp, $gp, %lo(_GLOBAL_OFFSET_TABLE_)

Here, the upper 16 bits of the $gp register are set with the “high” 16 bits of the value assigned to the special _GLOBAL_OFFSET_TABLE_ symbol, which provides the address of the special .got section defined in the linker script. This value is then combined using a logical “or” operation with the “low” 16 bits of the symbol’s value.

(I’m sure that anyone reading this far will already know that MIPS instructions are fixed length and are the same length as the address being loaded here, so it isn’t possible to fit the address into the load instruction. So each half of the address has to be loaded separately by different instructions. If you look at the assembled output of an assembly language program employing the “li” instruction, you will see that the assembler breaks this instruction down into “lui” and “ori” if it needs to. Note well that you cannot rely on the “la” instruction to load the special symbol into $gp precisely because it relies on $gp having already been set up.)

Rounding Off

I could probably go on about MIPS initialisation rituals, setting up a stack for function calls, and so on, but that isn’t really what this article is about. My intention here is to leave enough clues and reminders for other people in a similar position and perhaps even my future self.

Even though I have some familiarity with the MIPS architecture, I suppose you might be wondering why I am not evaluating more open hardware platforms. I admit that it is partly laziness: I could get into doing FPGA-related stuff and deploy open microcontroller cores, maybe even combining them with different peripheral circuits, but that would be a longer project with a lot of familiarisation involved, plus I would have to choose carefully to get a product supported by Free Software. It is also cheapness: I could have ordered a HiFive1 board and started experimenting with the RISC-V architecture, but that board seems expensive for what it offers, at least once you disregard the pioneering aspects of the product and focus on the applications of interest.

So, for now, I intend to move slowly forwards and gain experiences with an existing platform. In my next article on this topic, I hope to present some of the things I have managed to achieve with the PIC32 and a selection of different components and technologies.

free software - Bits of Freedom: Containerising the FSFE, aiming for Kubernetes & 2020

May 17 2017

DanielPocock.com - fsfe: Hacking the food chain in Switzerland

A group has recently been formed on Meetup seeking to build a food computer in Zurich. The initial meeting is planned for 6:30pm on 20 June 2017 at ETH, (Zurich Centre/Zentrum, Rämistrasse 101).

The question of food security underlies many of the world's problems today. In wealthier nations, we are being called upon to trust a highly opaque supply chain and our choices are limited to those things that major supermarket chains are willing to stock. A huge transport and storage apparatus adds to the cost and CO2 emissions and detracts from the nutritional value of the produce that reaches our plates. In recent times, these problems have been highlighted by the horsemeat scandal, the Guacapocalypse and the British Hummus crisis.

One interesting initiative to create transparency and encourage diversity in our diets is the Open Agriculture (OpenAg) Initiative from MIT, summarised in this TED video from Caleb Harper. The food produced is healthier and fresher than anything you might find in a supermarket and has no exposure to pesticides.

An open source approach to food

An interesting aspect of this project is the promise of an open source approach. The project provides hardware plans, a a video of the build process, source code and the promise of sharing climate recipes (scripts) to replicate the climates of different regions, helping ensure it is always the season for your favour fruit or vegetable.

Do we need it?

Some people have commented on the cost of equipment and electricity. Carsten Agger recently blogged about permaculture as a cleaner alternative. While there are many places where people can take that approach, there are also many overpopulated regions and cities where it is not feasible. Some countries, like Japan, have an enormous population and previously productive farmland contaminated by industry, such as the Fukushima region. Growing our own food also has the potential to reduce food waste, as individual families and communities can grow what they need.

Whether it is essential or not, the food computer project also provides a powerful platform to educate people about food and climate issues and an exciting opportunity to take the free and open source philosophy into many more places in our local communities. The Zurich Meetup group has already received expressions of interest from a diverse group including professionals, researchers, students, hackers, sustainability activists and free software developers.

Next steps

People who want to form a group in their own region can look in the forum topic "Where are you building your Food Computer?" to find out if anybody has already expressed interest.

Which patterns from the free software world can help more people build more food computers? I've already suggested using Debian's live-wrapper to distribute a runnable ISO image that can boot from a USB stick, can you suggest other solutions like this?

Can you think of any free software events where you would like to see a talk or exhibit about this project? Please suggest them on the OpenAg forum.

There are many interesting resources about the food crisis, an interesting starting point is watching the documentary Food, Inc.

If you are in Switzerland, please consider attending the meeting on at 6:30pm on 20 June 2017 at ETH (Centre/Zentrum), Zurich.

One final thing to contemplate: if you are not hacking your own food supply, who is?

May 16 2017

DanielPocock.com - fsfe: Building an antenna and receiving ham and shortwave stations with SDR

In my previous blog on the topic of software defined radio (SDR), I provided a quickstart guide to using gqrx, GNU Radio and the RTL-SDR dongle to receive FM radio and the amateur 2 meter (VHF) band.

Using the same software configuration and the same RTL-SDR dongle, it is possible to add some extra components and receive ham radio and shortwave transmissions from around the world.

Here is the antenna setup from the successful SDR workshop at OSCAL'17 on 13 May:

After the workshop on Saturday, members of the OSCAL team successfully reconstructed the SDR and antenna at the Debian info booth on Sunday and a wide range of shortwave and ham signals were detected:

Here is a close-up look at the laptop, RTL-SDR dongle (above laptop), Ham-It-Up converter (above water bottle) and MFJ-971 ATU (on right):

Buying the parts

Component Purpose, Notes Price/link to source RTL-SDR dongle Converts radio signals (RF) into digital signals for reception through the USB port. It is essential to buy the dongles for SDR with TCXO, the generic RTL dongles for TV reception are not stable enough for anything other than TV. ~ € 25 Enamelled copper wire, 25 meters or more Loop antenna. Thicker wire provides better reception and is more suitable for transmitting (if you have a license) but it is heavier. The antenna I've demonstrated at recent events uses 1mm thick wire. ~ € 10 4 (or more) ceramic egg insulators Attach the antenna to string or rope. Smaller insulators are better as they are lighter and less expensive. ~ € 10 4:1 balun The actual ratio of the balun depends on the shape of the loop (square, rectangle or triangle) and the point where you attach the balun (middle, corner, etc). You may want to buy more than one balun, for example, a 4:1 balun and also a 1:1 balun to try alternative configurations. Make sure it is waterproof, has hooks for attaching a string or rope and an SO-239 socket. from € 20 5 meter RG-58 coaxial cable with male PL-259 plugs on both ends If using more than 5 meters or if you want to use higher frequencies above 30MHz, use thicker, heavier and more expensive cables like RG-213. The cable must be 50 ohm. ~ € 10 Antenna Tuning Unit (ATU) I've been using the MFJ-971 for portable use and demos because of the weight. There are even lighter and cheaper alternatives if you only need to receive. ~ € 20 for receive only or second hand PL-259 to SMA male pigtail, up to 50cm, RG58 Joins the ATU to the upconverter. Cable must be RG58 or another 50 ohm cable ~ € 5 Ham It Up v1.3 up-converter Mixes the HF signal with a signal from a local oscillator to create a new signal in the spectrum covered by the RTL-SDR dongle ~ € 40 SMA (male) to SMA (male) pigtail Join the up-converter to the RTL-SDR dongle ~ € 2 USB charger and USB type B cable Used for power to the up-converter. A spare USB mobile phone charge plug may be suitable. ~ € 5 String or rope For mounting the antenna. A ligher and cheaper string is better for portable use while a stronger and weather-resistent rope is better for a fixed installation. € 5

Building the antenna

There are numerous online calculators for measuring the amount of enamelled copper wire to cut.

For example, for a centre frequency of 14.2 MHz on the 20 meter amateur band, the antenna length is 21.336 meters.

Add an extra 24 cm (extra 12 cm on each end) for folding the wire through the hooks on the balun.

After cutting the wire, feed it through the egg insulators before attaching the wire to the balun.

Measure the extra 12 cm at each end of the wire and wrap some tape around there to make it easy to identify in future. Fold it, insert it into the hook on the balun and twist it around itself. Use between four to six twists.

Strip off approximately 0.5cm of the enamel on each end of the wire with a knife, sandpaper or some other tool.

Insert the exposed ends of the wire into the screw terminals and screw it firmly into place. Avoid turning the screw too tightly or it may break or snap the wire.

Insert string through the egg insulators and/or the middle hook on the balun and use the string to attach it to suitable support structures such as a building, posts or trees. Try to keep it at least two meters from any structure. Maximizing the surface area of the loop improves the performance: a circle is an ideal shape, but a square or 4:3 rectangle will work well too.

For optimal performance, if you imagine the loop is on a two-dimensional plane, the first couple of meters of feedline leaving the antenna should be on the plane too and at a right angle to the edge of the antenna.

Join all the other components together using the coaxial cables.

Configuring gqrx for the up-converter and shortwave signals

Inspect the up-converter carefully. Look for the crystal and find the frequency written on the side of it. The frequency written on the specification sheet or web site may be wrong so looking at the crystal itself is the best way to be certain. On my Ham It Up, I found a crystal with 125.000 written on it, this is 125 MHz.

Launch gqrx, go to the File menu and select I/O devices. Change the LNB LO value to match the crystal frequency on the up-converter, with a minus sign. For my Ham It Up, I use the LNB LO value -125.000000 MHz.

Click OK to close the I/O devices window.

On the Input Controls tab, make sure Hardware AGC is enabled.

On the Receiver options tab, change the Mode value. Commercial shortwave broadcasts use AM and amateur transmission use single sideband: by convention, LSB is used for signals below 10MHz and USB is used for signals above 10MHz. To start exploring the 20 meter amateur band around 14.2 MHz, for example, use USB.

In the top of the window, enter the frequency, for example, 14.200 000 MHz.

Now choose the FFT Settings tab and adjust the Freq zoom slider. Zoom until the width of the display is about 100 kHZ, for example, from 14.15 on the left to 14.25 on the right.

Click the Play icon at the top left to start receiving. You may hear white noise. If you hear nothing, check the computer's volume controls, move the Gain slider (bottom right) to the maximum position and then lower the Squelch value on the Receiver options tab until you hear the white noise or a transmission.

Adjust the Antenna Tuner knobs

Now that gqrx is running, it is time to adjust the knobs on the antenna tuner (ATU). Reception improves dramatically when it is tuned correctly. Exact instructions depend on the type of ATU you have purchased, here I present instructions for the MFJ-971 that I have been using.

Turn the TRANSMITTER and ANTENNA knobs to the 12 o'clock position and leave them like that. Turn the INDUCTANCE knob while looking at the signals in the gqrx window. When you find the best position, the signal strength displayed on the screen will appear to increase (the animated white line should appear to move upwards and maybe some peaks will appear in the line).

When you feel you have found the best position for the INDUCTANCE knob, leave it in that position and begin turning the ANTENNA knob clockwise looking for any increase in signal strength on the chart. When you feel that is correct, begin turning the TRANSMITTER knob.

Listening to a transmission

At this point, if you are lucky, some transmissions may be visible on the gqrx screen. They will appear as darker colours in the waterfall chart. Try clicking on one of them, the vertical red line will jump to that position. For a USB transmission, try to place the vertical red line at the left hand side of the signal. Try dragging the vertical red line or changing the frequency value at the top of the screen by 100 Hz at a time until the station is tuned as well as possible.

Try and listen to the transmission and identify the station. Commercial shortwave broadcasts will usually identify themselves from time to time. Amateur transmissions will usually include a callsign spoken in the phonetic alphabet. For example, if you hear "CQ, this is Victor Kilo 3 Tango Quebec Romeo" then the station is VK3TQR. You may want to note down the callsign, time, frequency and mode in your log book. You may also find information about the callsign in a search engine.

The video demonstrates reception of a transmission from another country, can you identify the station's callsign and find his location?

<video controls="" height="480" width="640">
<source src="https://video.danielpocock.com/oscal_demo3.mp4" type="video/mp4">
Your browser does not support the video tag.

If you have questions about this topic, please come and ask on the Debian Hams mailing list. The gqrx package is also available in Fedora and Ubuntu but it is known to crash on startup in Ubuntu 17.04. Users of other distributions may also want to try the Debian Ham Blend bootable ISO live image as a quick and easy way to get started.

May 12 2017

agger's Free Software blog: Talk on nature and permaculture


Talk given at the technoshamanism encounter in Aarhus, November 22, 2014.

I wish to talk about the Danish nature, and I’d like to start by showing
you some pictures. The first pictures I’m going to show you were taken
by the writer and photogropher Rune Engelbreth Larsen who has been
working a lot on documenting the Danish nature as it is today.

Photo (c) Rune Engelbreth LarsenWhat you se here is of course a Danish forest with a lake, and if you
look at these pictures (which Rune Engelbreth has been dedicating
several years of his life to taking, making excursions and going to the
most remote corners of Danish nature) you’ll find that
many places in Denmark we actually have a rich nature
with very beautiful places, e.g. like this.

Photo (c) Rune Engelbreth LarsenSo if you visit a place like this and many others, you’d be forgiven for
thinking that if there are so many wonderful places here in Denmark
where you can go to meditate and enjoy the nature, we must have a
beautiful and very diverse nature in this country. And it’s true -
you can indeed find places where you can
take such pictures. You can find many such places, also e.g. like this:

Photo (c) Rune Engelbreth LarsenThese examples of Danish nature are places with a really peaceful
atmosphere. They are very green and very nice, usually with very
comfortable temperatures. If you visit some of the best places in Danish
nature, you’ll find that these are places that you can really feel a
part of, where you can relax and be happy and feel a strong sense of
belonging. As a Danish person I grew up with this kind of nature, and
I really feel at home in these tranquil spots. This is a landscape to
which I can really connect, spiritually.

Photo (c) Rune Engelbreth Larsen

In the photo above, you see one of our great oak forests, in Langaa.
Originally, Danish forests consisted mainly of oak trees; today, species
like beech, birch and fir are much more common. We don’t really
know why this forest has been conserved as oak, but in pre-Christian
times oak trees were connected to the ancient Nordic religion – the oak
is the tree of Thor. Thus, a very old oak forest like this one can also
help us connect to that part of our ancestral history.

However, if you look at other parts of the Danish landscape, a very
different picture emerges. The following pictures were all taken during a
walk that I took one day several years ago when I was living in
Solbjerg, a small town south of Aarhus. I walked around the lake called
Solbjerg Sø in one end and Stilling Sø in the other. There are, as you
can see, nice pastures along its banks that you can follow.

The trees that you see in the background are a small forest that’s also
quite beautiful once you get inside.

From there, you can go up a hill road and get a good view of the
landscape (click to see the full panorama image).

You have the hillside, the trees in the background, the
lake, and it’s all very nice, but: as pleasant as all of this might
appear, these pictures are actually lying. The first pictures I chose to
illustrate this article are truthful enough – they really do show some
very nice spots of Danish nature. But the pictures from my walk are
lying, all of them. Solbjerg Sø is situated in a completely agricultural
district, and the only reason there are green pastures at its banks is
that it’s too humid, so they can’t get their tractors all the way down
there. The trees at their edge form a hedge, and the corn fields start
right on the other side. In the autumn and winter, the pastures are
flooded and it’s nearly impossible to walk there. This is partly because
the lake’s water level is rising slightly, but mostly it’s because of
the drainage pipes that carry water from the fields on the hillside,
making them more tractor- and corn-friendly. This drainage water will
also carry excess fertilizer and pesticides which are sprayed on the
crop, all of which are flowing directly onto the pastures on their way
to the lake.

If you enter the forest on the other side of the lake – as I did that
day, since I walked all the way around the lake as close to the banks as
possible – you’ll se that it has been cut back and replaced with corn
fields as far down the slope as possible, right down to the point where
the ground becomes so humid that no matter what they do and how many
draining pipes they put in the ground, they still won’t be able to till
it and thus they can’t grow corn on it. The hillside is drained as far
down as possible, and the water from the fields is conducted to that
wood and makes it much too humid, creating a very bad climate in the
wood. The water from the fields is polluted with nutrients from the
chemical fertilizers used for the corn. This means that the wood by the
lake is overgrown with nettles, and otherwise the biological diversity
is nil. The high humidity does make it an ideal breeding place for
mosquitoes, though. As idyllic as that wood may look at a distance, it
really isn’t a nice place.

The area around Solbjerg Sø is a good example of a very idyllic but
completely destroyed Danish landscape. And this is the real state of
most of Danish nature these days.

On the other hand, if you look at the picture below, you’ll see a
typical Danish landscape. If you look at an aerial photo, you’ll find
that this is what Denmark is: Corn fields. You’ll see some tiny patches
of green dispersed in the landscape. These are forests that they haven’t
felled yet and some very few bogs that they were unable to drain
enough to sow corn.

These landscapes are really dreary. What few people know is that Denmark
is the most heavily farmed country in Europe and one of the most heavily
farmed – and most destroyed – countries in the world.

My assessment of the situation is that, financially speaking, we’re
doing all of this for nothing. The most important crop on Danish corn
fields is barley, which is mainly grown in order to feed pigs and other
livestock. The real reason for having all these corn fields is that we
have to feed the pigs in the industrial pig farms so that we can export a
lot of pork meat.

Meanwhile, in these pig factories the pigs are being overmedicated. All
of them are usually treated preventively with antibiotics. This is
creating new antibiotic-resistant bacteria which may also infect humans.
Right now, there’s a bacterium originating in pig farms called MRSA
(Methicillin-resistant Staphylococcus Aureus) which is actually killing
people. In Denmark, four people died of MRSA infection from 2012 to
2014. In order to protect the pig farmers from “harm” (e.g., in the
form of consumer boykot), the government is keeping the outbreak
locations secret. This means that school classes may and do visit farms
with outbreaks of this deadly disease. The teachers are not allowed to
make a informed decision not to visit an infected farm, since that
information is confidential.

So this is the dilemma: Our agricultural production is destroying Danish
nature and undermining the health of future generations by creating
penicillin-resistant strains of bacteria which are already lethal in
some cases. At the same time, we don’t really have to produce food in
that way, because we don’t need to use all this space on corn fields in
order to feed ourselves. Most of our agricultural production is
exported. Of course, one might claim that we produce all this grain and
pork meat for sound economic reasons. The agriculture lobby is always
emphasizing this claim.

But the fact is that agriculture as practised in Denmark is not
financially sustainable. The only reason that Danish farmers survive
economically is that they’re heavily subsidized. If it weren’t for
subsidies, they would quickly go bankrupt. In other words, the present
overfarming is pointless. We’re sacrificing our country’s nature and
biological diversity to the very short-sighted interests of one single
profession whose productions methods are not even economically viable.

This is the first of the dilemmas I want to address: We’re destroying
our country in order to produce far more food than we need, and society
as a whole is losing money because of it. Meanwhile, there are many
environmentally and economically sustainable ways to produce food.

A related dilemma is that today’s consumer society is moving faster and
faster. This means that we as people are constantly pushed harder and
harder to perform. You have to do well in school, you have to get good
grades in secondary school, you have to work hard at the university, you
have to get a good exam. Once you get out you have to get a good job.
You’ll often have to be there at nine o’clock in the morning and stay
until five. In most jobs, you must be there during these hours or
you’ll get fired. If you get fired and you’re eligible for unemployment
benefit, society has all sorts of rules so you have to go to meetings,
attend “courses” and turn up at eight or nine o’clock in the morning
anyway. Only now you won’t get paid more than you need to barely keep
you from starving, so you better find a job soon! In other words, our
society isn’t free at all: We’re forced to become parts of consumer
society and we’re not actually free to choose not to be a part of it.

One way in which we could become more free to choose how to live would be
to become less dependent on money.

I’d like to propose that the two problems I’ve described right now – the
destruction of nature by unsustainable and unnecessary food production
and modern society’s endemic lack of personal freedom – have one single
solution. We, as individuals, can obtain the means to sustain ourselves
and each other without having to work for money in the way we do now.
It is theoretically possible for us to create a sustainable life and a
sustainable production of food and other necessities in such a way that
our society is and remains in balance with nature.

If we were to organize our food production according to the principles
of permaculture; that is, if we were to produce what we need in forest
gardens and carefully designed multicultures, we could avoid the use of
chemical fertilizers, pesticides and fossil fuels. In permaculture you
normally design your crops to balance and suplement each other, so you
don’t need to supply fertilizer – a careful selection of edible
perennials can help provide the nutrients they need and keep each
other’s pests and diseases away. The only monoculture crop we really
have to cultivate is wheat, since people do want bread. In order to
produce bread, we need about 500 square meters (organically farmed) for
each person, and that’s nothing compared to the area we’re currently
using for agriculture. Apart from that, we could organize our
agriculture entirely according to permaculture principles. We could
easily do without agriculture as an export industry, since it’s already
not actually making us any money, since it’s only profitable because
it’s subsidized. Giving up on the pig production, we could shrink the
agricultural land to a fraction of what it is today. Instead of using
60% of the country’s area for agriculture, we could feed ourselves with
just 10% of the space; and we could re-establish nature in the rest of

That’s what we could do as a society if we want to transition to
something more sustainable. But we can also do something as individuals.
If we have the option we might, of course, shoot our proverbial uncle
in America and inherit all his millions – but most of us don’t have that
option. On the other hand, if we were to get cheap access to a garden -
I was lucky, as I was able to buy a house cheaply and I’m now building a
permaculture project there – we could put in a modest amount of work for
some years and afterwards we’d be able to get most of the food that we
need from that garden. It’s estimated that if you want to be
self-sufficient with food, you could get by with 400 square meters per
person, farmed very intensively. When I took my Permculture Design
Certificate, our teacher said that he wouldn’t try to do that himself -
he’d probably go for 800. But that’s still a moderately sized garden.
People could form communities with shared gardens – share the access,
share the cost, share the work, share the food – and you could have
communities of people who sustain themselves without ever spending any
money on food. So this transition is not just something we could do as
a society – we could do it ourselves and become vastly more independent
of having a job and being a part of consumer society. I’m not saying
this is the whole solution – there’s still some dependencies like how to
get the land in the first place – but it could be an important start for
many of us.

If you’re consuming pigs which have been produced by industrial
agriculture, then you’re also paying to have their fodder moved around,
you’re paying to have your fertilizers transported from abroad, you’re
paying to sustain a lot of carbon emissions. If you build a forest
garden in your own garden or help build a forest garden on a shared area
so that afterwards you can get most of your food from that forest
garden, you’re actually not emitting any carbon, you’re storing it. A
growing forest garden is actively removing carbon from the atmosphere
and storing it in the trees, in the roots beneath the ground and in all
the plants constantly growing in that garden. Calculations have been
made which show that if – and that’s a very big IF – we were to
transition to get a large part of our food from forest gardens and
sourcing that produce locally instead of having everything driven and
flown around, we would be able to take enough carbon out of the Earth’s
atmosphere to make a difference regardint the climate changes. I’m not
saying that this is going to happen because obviously it won’t, but we
could theoretically avert some of the dangers of climate change that

Maybe we can’t save the world, but we can make ourselves more
independent and our lives freer and more fulfilling – and in doing so we
can at least try to contribute as little as possible to our own

DanielPocock.com - fsfe: Thank you to the OSCAL team

The welcome gift deserves its own blog post. If you want to know what is inside, I hope to see you at OSCAL'17.

DanielPocock.com - fsfe: Kamailio World and FSFE team visit, Tirana arrival

This week I've been thrilled to be in Berlin for Kamailio World 2017, one of the highlights of the SIP, VoIP and telephony enthusiast's calendar. It is an event that reaches far beyond Kamailio and is well attended by leaders of many of the well known free software projects in this space.

HOMER 6 is coming

Alexandr Dubovikov gave me a sneak peek of the new version of the HOMER SIP capture framework for gathering, storing and analyzing messages in a SIP network.

exploring HOMER 6 with Alexandr Dubovikov at Kamailio World 2017

Visiting the FSFE team in Berlin

Having recently joined the FSFE's General Assembly as the fellowship representative, I've been keen to get to know more about the organization. My visit to the FSFE office involved a wide-ranging discussion with Erik Albers about the fellowship program and FSFE in general.

discussing the Fellowship program with Erik Albers

Steak and SDR night

After a hard day of SIP hacking and a long afternoon at Kamailio World's open bar, a developer needs a decent meal and something previously unseen to hack on. A group of us settled at Escados, Alexanderplatz where my SDR kit emerged from my bag and other Debian users found out how easy it is to apt install the packages, attach the dongle and explore the radio spectrum.

playing with SDR after dinner

Next stop OSCAL'17, Tirana

Having left Berlin, I'm now in Tirana, Albania where I'll give an SDR workshop and Free-RTC talk at OSCAL'17. The weather forecast is between 26 - 28 degrees celsius, the food is great and the weekend's schedule is full of interesting talks and workshops. The organizing team have already made me feel very welcome here, meeting me at the airport and leaving a very generous basket of gifts in my hotel room. OSCAL has emerged as a significant annual event in the free software world and if it's too late for you to come this year, don't miss it in 2018.

OSCAL'17 banner

May 11 2017

Evaggelos Balaskas - System Engineer: time offset and nagios nrpe check

What is the time?

Time offset is the amount of time that is off (or drift) from a specific value. In Linux systems, date is been calculating from the beginning of time. That is 00:00:00 1 January 1970 or as it called Unix Time and systems define date (time) as the number of seconds that have elapsed from 01.01.1970.

It is so important that even a few seconds off can cause tremendous disaster in data centers and applications.

Network Time

To avoid problems with time, systems must and should synchronize their time over the Internet every now and then. This is being done by asking a central NTP server via Network Time Protocol. The most common scenario for infrastructures is to have one or two NTP servers and then all the systems inside this infrastructure can synchronize their time from those machines.

Nagios - NRPE

In my case, I have a centralized NTP Daemon that runs on the Nagios Linux machine. That gives me the opportunity to check the EPOCH time of any system in my infrastructure against the time that the Nagios Server has.

Nagios Check

This is the script I am using:

# ebal, Thu, 11 May 2017 12:08:50 +0000


# seconds
OFFSET=$( echo $(( $(date -d 'now ' +%s) - ${TIME} )) | sed -e 's#-##g' )

if [ "${OFFSET}" -lt "${WARN}" ]; then
        echo "OK"
        exit 0
elif [ "${OFFSET}" -ge "${CRIT}" ]; then
        echo "CRITICAL- ${OFFSET}"
        exit 2
elif [ "${OFFSET}" -lt "${CRIT}" ]; then
        echo "WARNING- ${OFFSET}"
        exit 1
        echo "UNKNOWN- ${OFFSET}"
        exit 3

In a nutshell the script gets as the first argument an epoch time and calculate the diff between it’s own epoch time and that.


./check_time_offset $(date -d 'now + 1 min' +%s)

The output is this:


Nrpe Configuration

This is the configuration for nrpe to run the check_time_offset

# tail -1 /etc/nrpe.d/time_offset.cfg

command[check_time_offset]=/usr/lib64/nagios/plugins/check_time_offset $ARG1$

Nagios Configuration

and this is my nagios configuration setup to use a remote nrpe :

define service{
        use                             service-critical
        hostgroup_name                  lnxserver01
        service_description             time_offset
        check_command                   check_nrpe!check_time_offset!$TIMET$

Take a minute to observer a little better the nrpe command.



I was having problems passing the nagios epoch time as an argument on the definition of the above service.

Testing the nrpe command as below, I was getting the results I was looking for:

./check_nrpe -H lnxserver01 -c check_time_offset -a $(date -d 'now + 6 sec' +%s)

But is there a way to pass as a nagios argument the output of a command ?

  • No

A dear colleague of mine mentioned nagios macros:

Standard Macros in Nagios

$TIMET$     Current time stamp in time_t format (seconds since the UNIX epoch)

Perfect !!!

Tag(s): nagios, nrpe, ntp, time

May 10 2017

vanitasvitae's blog » englisch: GSoC: First week of community bonding

The first week of community bonding is nearly over and already it’s quite an experience for me. Me and Alameyo were very nicely welcomed by members of the igniterealtime project which really took care of making us able to jump right into the project.

I spent most of the time making myself more familiar with the Smack codebase by working on my OMEMO pull request, which comes closer and closer to a mergeable state. Currently I’m working together with my mentor Flo to make the API of the module as easy to use as possible. I learned the hard way, that singletons and generics are a pain when used together, but in the end it worked out quite well I’d say, although there is still some work left to do.

During the week, I also came across some bugs in XMPP server software, which were a great opportunity to come in contact with other developers outside of the igniterealtime project. Everybody was nice and helpful. Its really pleasant to work with the community.

Lastly, I also got in touch with some other GSoC students, which is super fun. I really enjoy meeting people from other countries and it turns out, that everybody puts their pants on the same way.

I’ll post updates roughly on a weekly basis to document the course of events. Untill then :)

Happy Hacking!

May 08 2017

DanielPocock.com - fsfe: Visiting Kamailio World (Sold Out) and OSCAL'17

This week I'm visiting Kamailio World (8-10 May, Berlin) and OSCAL'17 (13-14 May, Tirana).

Kamailio World

Kamailio World features a range of talks about developing and using SIP and telephony applications and offers many opportunities for SIP developers, WebRTC developers, network operators and users to interact. Wednesday, at midday, there is a Dangerous Demos session where cutting edge innovations will make their first (and potentially last) appearance.

Daniel Pocock and Daniel-Constantin Mierla at Kamailio World, Berlin, 2017

OSCAL'17, Tirana

OSCAL'17 is an event that has grown dramatically in recent years and is expecting hundreds of free software users and developers, including many international guests, to converge on Tirana, Albania this weekend.

On Saturday I'll be giving a workshop about the Debian Hams project and Software Defined Radio. On Sunday I'll give a talk about Free Real-time Communications (RTC) and the alternatives to systems like Skype, Whatsapp, Viber and Facebook.

Henri Bergius: Edit as project and Flowhub live mode

In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes.

Live mode

Flowhub’s live mode is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it.

Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device:

<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/EdgeSDFd9p0" width="560"></iframe>

In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code.

Edit as project

To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project.

MsgFlo in live mode

In nutshell:

  • Live mode provides a read-only view to a running system
  • Project mode allows live programming a full project including its graphs and components
  • Edit as project can be used to turn a live mode session into an editable project
  • Projects can be synchronized with GitHub

Edit as project also works with the example programs that ship with Flowhub.

May 07 2017

Evaggelos Balaskas - System Engineer: How a slow disk affects your system

The problem

The last couple weeks, a backup server I am managing is failing to make backups!

The backup procedure (a script via cron daemon) is to rsync data from a primary server to it’s /backup directory. I was getting cron errors via email, informing me that the previous rsync script hasnt already finished when the new one was starting (by checking a lock file). This was strange as the time duration is 12hours. 12 hours werent enough to perform a ~200M data transfer over a 100Mb/s network port. That was really strange.

This is the second time in less than a year that this server is making problems. A couple months ago I had to remove a faulty disk from the software raid setup and check the system again. My notes on the matter, can be found here:


Identify the problem

So let us start to identify the problem. A slow rsync can mean a lot of things, especially over ssh. Replacing network cables, viewing dmesg messages, rebooting servers or even changing the filesystem werent changing any things for the better. Time to move on the disks.

Manage and Monitor software RAID devices

On this server, I use raid5 with four hard disks:

# mdadm --verbose --detail /dev/md0

        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 2929893888 (2794.16 GiB 3000.21 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun May  7 11:00:32 2017
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ServerTwo:0  (local to host ServerTwo)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 10496

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       6       8       48        2      active sync   /dev/sdd
       5       8        0        3      active sync   /dev/sda

View hardware parameters of hard disk drive

aka test the hard disks:

# hdparm -Tt /dev/sda

 Timing cached reads:   2490 MB in  2.00 seconds = 1245.06 MB/sec
 Timing buffered disk reads: 580 MB in  3.01 seconds = 192.93 MB/sec

# hdparm -Tt /dev/sdb

 Timing cached reads:   2520 MB in  2.00 seconds = 1259.76 MB/sec
 Timing buffered disk reads: 610 MB in  3.00 seconds = 203.07 MB/sec

# hdparm -Tt /dev/sdc

 Timing cached reads:   2512 MB in  2.00 seconds = 1255.43 MB/sec
 Timing buffered disk reads: 570 MB in  3.01 seconds = 189.60 MB/sec

# hdparm -Tt /dev/sdd

 Timing cached reads:     2 MB in  7.19 seconds = 285.00 kB/sec
 Timing buffered disk reads:   2 MB in  5.73 seconds = 357.18 kB/sec

Root Cause

Seems that one of the disks (/dev/sdd) in raid5 setup, is not performing as well as the others. The same hard disk had a problem a few months ago.

What I did the previous time, was to remove the disk, reformatting it in Low Level Format and add it again in the same setup. The system rebuild the raid5 and after 24hours everything was performing fine.

However the same hard disk seems that still has some issues . Now it is time for me to remove it and find a replacement disk.

Remove Faulty disk

I need to manually fail and then remove the faulty disk from the raid setup.

Failing the disk

Failing the disk manually, means that mdadm is not recognizing the disk as failed (as it did previously). I need to tell mdadm that this specific disk is a faulty one:

# mdadm --manage /dev/md0 --fail /dev/sdd
mdadm: set /dev/sdd faulty in /dev/md0

Removing the disk

now it is time to remove the faulty disk from our raid setup:

# mdadm --manage /dev/md0 --remove  /dev/sdd
mdadm: hot removed /dev/sdd from /dev/md0

Show details

# mdadm --verbose --detail /dev/md0

        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 2929893888 (2794.16 GiB 3000.21 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun May  7 11:08:44 2017
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ServerTwo:0  (local to host ServerTwo)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 10499

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       4       0        0        4      removed
       5       8        0        3      active sync   /dev/sda

Mounting the Backup

Now it’s time to re-mount the backup directory and re-run the rsync script

mount /backup/

and run the rsync with verbose and progress parameters to review the status of syncing

/usr/bin/rsync -zravxP --safe-links --delete-before --partial --protect-args -e ssh /backup/

Everything seems ok.

A replacement order has already been placed.

Rsync times manage to hit ~ 10.27MB/s again!

Tag(s): linux, mdadm, md0

May 06 2017

vanitasvitae's blog » englisch: Using Emoji for fingerprint verification

The messaging app Telegram recently introduced end-to-end encrypted voice calls. As most of you probably know, encryption without verification is pretty useless since there is the risk of man-in-the-middle attacks. I don’t want to get too much into details about this. The point I want to make is, that you should verify your partners fingerprint (hash of the used key) in order to be secure.

The interesting part of Telegrams new feature is the way they verify fingerprints. Traditionally you are presented with a String of (typically hexadecimal – 0-9,A-F) characters. In the case of Conversations the fingerprint are 64 hexadecimal characters. Telegram on the other hand introduced the way of displaying 4 out of a set of 333 emojis (1). Note that this is only used to verify that the current voice call is secure. The next call would have a different fingerprint, so keep in mind, that we are talking about two different use cases here.

Still, how do those two methods compare? Could we use emoji in conversations to verify the fingerprint of identity keys?

With telegrams emoji encoding, there are 333⁴ = 12.296.370.321 possible hash values. This is tiny compared to the number of possibilities with the conventional fingerprint encoding (64 hex characters), which sum up to 16⁶⁴ = 115.792.089.237.316.195.423.570.985.008.687.907.853.269.984.665.640.564.039.457.584.007.913.129.639.936 unique hash values. This is far more secure than the system used by telegram. To be precise, it is 9.416.769.844.639.765.662. times more secure (theoretically).

But could we use emoji in eg. Conversations?

Lets say, we want to use emojis for fingerprint verification without trading away security. We’d have to display 31 emojis in order to be as secure as displaying 64 hex chars. Since most people are more familiar with numbers and the letters A-F, I doubt that this brings any benefits (we just cut the length of the string in half).

But what if we chose from a bigger set of emojis?

Lets say we want the fingerprint to be as short as the one in telegram (4 characters), but have the same security properties as the conventional 64 digit hex string. In order to encode the same amount of information in 4 symbols as we could in 64 hex characters, we’d have to use a pool of 18.446.744.073.709.551.616 symbols. Unfortunatelly there aren’t so many characters, let alone emojis.

But what would be the middle ground?

If we want our fingerprint to be 16 characters long, our character pool would be 65536 symbols. Thats the whole unicode space. Since there are many unicode characters that look alike and there are also a lot of “holes” in the unicode space, there are fewer usable characters.

In conclusion, it is not really possible/valuable to use emojis for fingerprint representation without trading away security.

(1): https://core.telegram.org/techfaq#q-how-are-voice-calls-authenticated

May 02 2017

agger's Free Software blog: Installing Baobáxia at the II International Festival of Technoshamanism

A scene from a short film created as part of the festival's video workshop.

On November 9 2016, I and my son Johannes left Denmark for Brazil in order to co-organize and attend the II International Festival of Technoshamanism. You can read more about the background for this festival at the technoshamanism site as well as in previous posts on this blog.

Each participant in the festival was expected to propose an activity, and as one of the organizers I was no exception. The II International Festival of Technoshamanism took place in the Pataxó village Pará in the extreme south of the Brazilian state of Bahia, and my proposal was to install a working node (mucúa) of the Baobáxia system in the village. The purpose of this node is to act as an archive of the indigenous cultural production, a way yo protect and salvage the indigenous culture in electronic form for decades to come, and a way for the Pataxó to do so with complete ownership over the infrastructure as well as the content – independently of corporate and proprietary content-sharing sites such as Facebook and YouTube.

Who are the Pataxó?

Well, I’ve written about them before, after I participated in the first technoshamanism festival in 2014. The Pataxó are an indigenous people (what we sometimes call “Indians”, indeed they sometimes call themselves that) who live near the sea in the extreme south of the Brazilian state of Bahia. The Aldeia Pará (Pará Village) is located in what the Pataxó call their origin or Aldeia Mãe, the area from which they were all expelled in the massacre of 1951. It is situated far from everything in the middle of an indigenous reserve comprising some 8500 hectares and is home to some 69 families. The neighboring village of Barra Velha is located 5 km from Pará and has a population of about 400 families.

The Pataxó are an intelligent and open-minded people, and though they have not have much exposure to computers, they are very conscious of the potential of modern technology in the struggle for their culture and their land, which has been a constant factor in Pataxó life for several decades – not least given that many of their villages are threatened by expulsion – in October 2016, the village of Aratikum some 100 kilometers north of Pará was actually razed to the ground by the police in the service of local land owners.

Opening ritual in Akurinã Pataxó's healing space.

The Pataxó’s motivation for hosting the event was to be able to exchange knowledge – to learn something from all us urban hackers and activists with our experience in free software and art and social movements, and to teach us something in the bargain, if we so desired. Each participant would propose an activity – a workshop, an art installation, a ritual, a performance, a talk – and would be free to participate in the other proposed activities. Among the activities proposed by the Pataxó were the festival’s opening and closing rituals, a healing tent, traditional Pataxó cooking, traditional body paint, fishing and hunting methods, the cultivation of manioc and manufacturing of manioc flour, musical sessions and the jogos indigenas, the indigenous sports games which took the form of a competition between Indians and non-Indians.

As I said, my proposal was to install a node of Baobáxia and – just as, or even more, important – give workshops in how to use and maintain the system (running on a dedicated Debian GNU/Linux server) themselves so it could be useful also after we left.

Technoshamanism, ancestrality and the Destructor

But if the Pataxó’s motivation for hosting the event was to make friends and allies and gain new knowledge, then what was our motivation for organizing it? And by “we” I mean a large group of people in the technoshamanism network, mainly but (obviously enough) not exclusively Brazilian: Fabi Borges, Jonatan Sola,  Sue Nhamandu, Rafel Frazão, Rodrigo Krul and too many more to mention, apart from the many participants who arrived at the festival with similar motives and gave many outstanding contributions?

One thing is the connection of technoshamanism with reconnection  - reconnecting with the Earth, reconnecting with the ancestral world views of the thousands of generations of people who lived close to the Earth in a mainly oral culture. The Pataxó live in a reserve where they can live well off the earth, they have a strong connection to their ancestral way of living as well as, quite literally, to their ancestors and other ancestral spirits, who often show up at their rituals.  As such, the Pataxó have 500 years of experience in dealing with European colonizers and the usurping civilization, and they have developed an immense skill in navigating this kind of pressure without losing neither their independence nor their traditional culture.

As opposed to that, the norm in our cities is that of disintegration, not least of community spirit and ancestral culture. Traditional songs and tales which might have been handed down in subtly changing ways for hundreds or thousands of years are replaced with comic books and cinema, which are replaced with endless children’s TV shows, which as we grow up are replaced with “breaking news”, X Factor and a host of even more diluted and inane TV shows. Culture ceases to be something we do ourselves and do together, neighborhoods cease to be communities and the cultural divide even splits up the families, so that we end up as disjoint individuals in a sea of strangers who can only struggle to recreate something vaguely resembling a genuine community. Sometimes, of course, as in the case of many successful free software projects, genuinely succeeding.

In his recent novel “Jerusalem“, the  writer and comic book author Alan Moore metaphorically describes this phenomenon, which he has experienced first hand in his home town of Northampton, as “the Destructor”. The Destructor was a garbage incinerator which for decades was actually and physically located in the poorest neighborhood in Northampton, reducing people’s life expectancy with at least ten years, its location a daily reminder to the inhabitants of the Boroughs of how little the rest of the city cared about them.

And yet The Boroughs was actually the oldest neighborhood in Northampton and home to a bafflingly rich, orally transmitted ancestral working class culture which was, after World War I and under the impression of the Russian revolution, deliberately crushed by city planning. According to Alan Moore, the policies which have disempowered modern Europeans by stripping us of our communities were deliberately inflicted. Moore describes the destructive effect on the communities with these words:

He saw a hundred old men and old women moved from the condemned homes where they’d raised their families, dumped in distant districts with nobody that they knew and failing to survive the transplant. By the dozen they keeled over on the well-lit stairs of their new houses; in the unfamiliar indoor toilets; onto their unprecedented fitted carpets; on the pillows of magnolia-painted bedrooms that they failed to wake to. Countless funerals fell into the Mayorhold’s fires, and furtive teenage love-affairs, and friendships between relocated children sent to different schools. Infants began to understand that they would probably now never marry the classmate the had been expecting to. All the connecting tissue, the affectations and associations, became cinders. (p. 731)

The Northampton neighborhood known as The Boroughs descended into complete misery and insecurity, containing the points of trade that “supplied the customers who drew the girls, who brought the pimps, who dealt the drugs, which bred the guns that shot the kids who lived in the house that crack built” (p. 691). But there’s a point in that – that kind of misery is very common in urbanized Brazil as well as in Europe, and a contact with people who still retain an orally transmitted culture and whose communities were never fragmented by the Destructor could teach us something about reconnecting, with the Earth and its spirits, with our natural spirituality and with true community.

The Festival area

As we arrived in Pará about November 14, our first job was to establish a good contact with the Pataxó, organize food for the event and start rigging the computers and other technical equipment.

As everybody else, we were camping in what was at first quite precarious conditions due to the heavy rainfall before the festival started.

Luckily, the Pataxó were very helpful and we managed to secure everything against the rain before the start of the festival on November 22.

Installing Baobáxia

Community Radio sending from the Pataxó Kijeme Cultural, home of the GNU/Linux computers

Before our arrival, the Pataxó had built a completely new house for cultural production, in which they had placed four stationary computers they had received from the reservation’s Fisherman’s Association which had originally got them from a government program. These four computers were quite old and had Windows installed, which we wanted to replace with GNU/Linux.

At first, our attempts at setting up the computers were haunted by technical difficulties. First of all, we were unable to get them to boot from USB drives, which meant we had to by burnable CDs or DVDs. When we got them, we realized they could not really boot from the DVDs either due to our images being 64 bit, and these trusty old computers were actually 32 bit. We could not use the Internet for troubleshooting since there was no Internet yet – it was supposed to arrive during  the week before the festival, but the roads wer closed due to the rain.

In the end, Pablo Vieira from the Assentamento Terravista near Ilhéus (with the microphone in the picture above) arrived, and as it turned out, he knows these computers very well; they can boot from USB if a rather obscure BIOS setting is enabled. In his pocket was a bootable USB with the most recent 32 bit Linux Mint, and everyone was happy and the computers were well prepared for the arrival of the Internet later that week.

The Internet arriving at Aldeia Pará. Pataxó warrior Txayhuã is painting festival organizer Fabi Borges while the operator's car has stopped at the new culture house. Half an hour later, there was Internet.

Vincenzo Tozzi from the Mocambos networks explains Baobáxia to village chief Ubiratã. Also listening are Pablo Vieira and Arapaty Pataxó.

I was not alone in the task of installing Baobáxia and giving workshops about it – Vincenzo Tozzi from the Mocambos network, Sicilian and founder of the Baobáxia project, joined the festival as well. Vince is a programmer and computer scientist and wrote a major part of the Baobáxia system himself, but really he’s a philosopher of networks with important insights in the potential of free software and offline digital communications, and his presence was an invaluable contribution to the festival.
Our two workshops in Baobáxia were a huge success, and especially the younger generation of the Pataxó showed a great interest in working with this technology. The Baobáxia node we installed is still active in the village and is still not connected to the Internet, but you can see the contents in its present degree of synchronization here.

What else was in the festival?

A lot of things.

Some very beautiful rituals:

And video workshops, radio workshops, capoeira, samba in the church in honor of Saint Benedito, seed exchange, agroforestry, construction of dry composting toilets, radio production, discussions about the pros and cons of ecoturism, and much, much more. I might do a followup post on that, in the meantime let it be said that the festival was a unique experience and I’m very happy to be one of the people who made it happen.

Free Software – Frank Karlitschek_: Nextcloud is about collaboration

Today the Nextcloud community released the Nextcloud 12 beta. The final release will be out later this month. This is a major new step forward. And it is also an interesting release because we are entering a new area for the product.
At the beginning, 7 years ago the focus was clearly file sync and share. Of course this term did not exist at the time or at least I didn’ know it. The task was to syncronize file between all your devices and share it with others.
This is obviously still the core of Nextcloud and Nextcloud 12 comes with all the usual iterative improvements. For example additional security hardening and improved two factor authentication, significant performance improvements especially for big installations, a ton of bugfixes and more powerful sharing features to make sharing easier and more powerful.

But it is clear that basic file sync and share is not enough anymore. People don’t want to see files popup or change on their devices without knowing what, why, when, who did this. They need ways to communicate and collaborate around the files that are shared. So the obvious next step is to add context, communication and collaboration features to the basic file sync and share.

Nextcloud already has added these areas. The focus of 12 was to combine them together into a powerful collaboratio and communication platform. Basically a fully open source and self hosted Office 365 or Google G Suite alternative.

So Nextcloud comes with powerful Calendar, Contacts and Email apps. There are apps for chat, tasks management, Kanban style planing, notes taking and many other things. The goal was to bring them all together. In Nextcloud 12 you can click on any avatar and directly start a chat conversation, start a video call via WebRTC or send an email. This is all grouped around the right context. So you can do this while sharing or editing a file. With the help of Collabora Online you can also do collaborative editing of office documents together.
There is a new contacts drop down in the main menu where you directly find your colleagues and friends and can interact with them. Of course you can easily find them, even if they are on a different Nextcloud server with the help of our new Lookup server and federated APIs. It is now super easy to switch between this apps with the help with the new apps navigation in the header.
The concepts of collaboration, communication and context also includes the Nextcloud mobile apps for iOS and Android. In both you can now see your activity stream to see who is doing what on the server. There are also new instant push notification to the phones if important events happen like someone left a comment to a file, send you a share invitation or invites you to a video call.

Of course this kind of integration around communication and collaboration should extend to other areas like your Desktop. This is why I’m super happy to see plugins like this KDE plugin to share directly from within your Plasma Desktop. Also the Nextcloud integration that the GNOME community is doing for calendar, contacts, maps and more is really awesome.

More information about Nextcloud 12 including some videos can be found here.

Let’s work together to create an open source alternative to the closed products from Microsoft or Google.

Everyone is invited to contribute to Nextcloud. Nextcloud is 100% open source without proprietary parts. Contributors don’t need to sign a contributor license agreement or give up any rights on their code..

The planing of the next release has already started so join us hereon github to contribute with ideas, code or anything else.

May 01 2017

Tarin Gamberini: Adoption of Free Software PDF readers in Italian Regional Public Administrations (fourth monitoring)

The following monitoring shows that, in the last semester, nine Italian Regions have reduced advertisement about proprietary PDF readers on their website, and that a Region has raised its support to Free Software PDF readers.

Continue reading →

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!