My browser is getting littered with Django references. Time to collect them together:
Tuesday, October 31. 2017
Django Training / Tutorials
NanoGUI
Something I need to integrate into some projects: NanoGUI:
is a minimalistic cross-platform widget library for OpenGL 3.x or higher. It supports automatic layout generation, stateful C++11 lambdas callbacks, a variety of useful widget types and Retina-capable rendering on Apple devices thanks to NanoVG by Mikko Mononen. Python bindings of all functionality are provided using pybind11.
Conferences And Presentations
Coincidental to my giving a talk at the beginning of December about 'The Network is the Computer, Security and Privacy for the Connected', I have started to see articles regarding best ways of delivering presentations and tools for developing presentations. I record them here for my reference:
- How to Give a Perfectly Adequate Conference Talk
- reveal.js: HTML Presentation Framework
- ReactJS based Presentation Library: Spectacle is a React.js based library for creating sleek presentations using JSX syntax that gives you the ability to live demo your code.
Building ZFS on Debian Stretch
Due to various licensing compatibility issues, which are described at What does it mean that ZFS is in Debian and On ZFS on Debian, source-only packages are available for ZFS on Debian Linux. Binaries need to be 'self-built'. Here is my method for building those binaries as packages.
I found some background information for building the packages in Debian bug #554843.
To start, add 'contrib' to /etc/apt/sources.list and run 'apt update'.
There are two dkms modules which need building: the ZFS kernel module, which depends upon the Solaris Porting Layer kernel module.
This process will need to be performed each time the kernel package gets updated or any of the related ZFS packages are updated. This process builds the kernel modules, and could be performed on a 'build machine', as various extra packages get installed to support the process:
apt install linux-headers-$(uname -r) apt install dpkg-dev fakeroot debhelper DEBIAN_FRONTEND=noninteractive apt-get -y --no-install-recommends install spl-dkms DEBIAN_FRONTEND=noninteractive apt-get -y --no-install-recommends install zfsutils-linux zfs-zed zfs-dkms
Packages can then be built and transported for installation on other machines:
dkms mkbmdeb spl -v 0.6.5.9 --dkmsframework framework.conf --binaries-only dkms mkbmdeb zfs -v 0.6.5.9 --dkmsframework framework.conf --binaries-only
Sunday, October 29. 2017
pi-hole in a Debian LXC container based upon Stretch
Some sites will generate queries to other domains for tracking purposes of various sorts. These queries tend to slow down the browser experience: a) the dns lookup itself takes time, then b) the round trip time to perform whatever nefarious action takes time.
To prevent some of these 'unneeded' or 'un-necessary' queries, it is possible to block them with a tool called pi-hole. It bills itself as "a black hole for internet advertisements".
It does indeed help speed up the browser experience.
The installation is relatively painless. I use a local package proxy using apt-cacher-ng, so I am recording my use of the changed sources list file, so I can remember it next time. In addition, the script doesn't seem to correctly configure php, so it fails once. I do a dummy install command with dnsmasq to get settings updated. Then run the script again, and it runs to completion.
sed -i 's_//deb_//<ip address>:3142/deb_' /etc/apt/sources.list sed -i 's_//sec_//<ip address>:3142/sec_' /etc/apt/sources.list cat /etc/apt/sources.list apt update apt install curl curl -sSL https://install.pi-hole.net | bash apt install dnsmasq curl -sSL https://install.pi-hole.net | bash
The end result is some text supplying a url for the administration interface, a password and a command to change the password:
pihole -a -p
A web page with Block Lists for Pi-Hole: CryptoAUSTRALIA's Favourite Block Lists
2018/01/09: Ad and tracking blocking dnsmasq-ready blocklists: (github.com)
2018/08/17 Domain Name Service Response Policy Zones (DNS RPZ) is a method that allows a nameserver administrator to overlay custom information on top of the global DNS to provide alternate responses to queries. It is currently implemented in the ISC BIND nameserver (9.8 or later). Another generic name for the DNS RPZ functionality is "DNS firewall".
- Project Tar
- No Listing - Poor Man's Greylisting
- spam trap - multi-threaded daemon that provides a RFC 2821 compliant SMTP service that always returns a 4xx soft error or 5xx hard error to the RCPT TO verb.
- Set up a SMTP decoy: I have used this one. Modify it to issue 5xx permanenet errors instead of 4xx retry errors.
-
http://smtptrapd.sourceforge.net/
- Install it on one of your hosts you have control over.
- *MOST IMPORTANT* firewall block it, only *allowing* the spammer's MX to talk to your trap server.
- Change your MX records to insert the SMTP trap host as your preferred MX ahead of your real MX.
- dns rate limit
open vswitch newsgroup notes
Bypassing OpenFlow controller and injecting flow table entries via a code call:
> Can someone refer to the code where a new flow is inserted to > openflow tables in the OVS, as a consequence action of a received > message from a controller? handle_flow_mod() in ofproto.c
Another similar request:
> My project was previously calling system commands like ‘ovs-ofctl add-flow’ directly from C, > but we would like to do this programmatically now, by calling into the openvswitch library. > After looking at this for a bit and what methods I would have to call, I’ve realized this is > non trivial, and I can’t find any easily exposed methods to add and delete flows. > Does anyone have an example anywhere of calling into openvswitch methods > directly to add/remove flows? So far the best way I can find is calling into > ovn/controller/ofctrl.h:ofctrl_add_flow(…), but this will require quite a bit of legwork. One way: There are plenty of controllers out there, including a couple in C - libfluid Another way: Here’s how I do it. It’s essentially the same as what the ‘ovs-ofctl add-flow’ command does under the hood. 1. Build a string formatted as the “ovs-ofctl” command would do, for example “table=1,cookie=0xdeadbeef,in_port=1,actions=resubmit(,2)” 2. Pass this to parser_ofp_flow_mod_str(). The parser will magically parse the string in to a rich struct. 3. Pass the struct to ofputil_encode_flow_mod(). This returns a struct with a serialized OpenFlow message buffer. 4. Send this buffer on the wire either via a socket you manage yourself or let OpenVswitch do it for you.
Added 2017/11/13, the second link shows a terrific example usage for ovs-dpctl
> I have a problem that when i use the ovs-dpctl to add a flow into datapath, > it occurs "ovs-dpctl: parsing flow key (Invalid argument)" > > example: > root@jlt:~# ovs-dpctl add-flow system@myDP "in_port(1),eth_type(0x800),ipv4 > (src=172.31.110.4,dst=172.31.110.5)" 2 > ovs-dpctl: parsing flow key (Invalid argument) You may be able to use 'dmesg' to see which key is missing. See the talk by Joe Stringer: Youtube And the blogpost (shameless plug for myself): Direct Kernel OVS Flow Programming
A follow up from Ben indicated though:
I believe that this particular error comes from the userspace parser, not the kernel.
Another addition on 2017/11/13:
> For us, newbies, examples are extremely valuable, more than a thousand > words. > If some kind soul has an example on how to insert a new interface (s1-eth3) > in a single switch (s1) with two interfaces (s1-eth2 and s1-eth3) please > share with me. If you run something like this: ovs-vsctl -vjsonrpc -- add-bond s1 s1-eth3 s1-eth2 s1-eth3 then you will see what ovs-vsctl does to insert such a bond, logged as the jsonrpc module.
2017/11/25 Addition 1:
ovs-save is a shell script, using `ovs-ofctl dump-flows br` to write flows into a file, and then use `ovs-ofctl add-flows br FILE` to add them.
2017/11/25 Addition 2:
The OpenFlow Spec does not directly provide the mechanism for the multi-path load balancing and also Ryu does not provide it. I guess it is depending on your application design...
But for the beginning, how about using the Group action(the "select" type)? The selection algorithm is depending on your switch though, it provides an easy way to do load balancing.
Please refer to the section "5.6.1 Group Types" in the OpenFlow Spec 1.3.5 for the details: https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-switch-v1.3.5.pdf
2017/12/07 dpctl: Support flush conntrack by 5-tuple
With this patch, "flush-conntrack" in ovs-dpctl and ovs-appctl accept a conntrack 5-tuple to delete the conntrack entry specified by the 5-tuple. For example, user can use the following command to flush a conntrack entry in zone 5.
$ ovs-dpctl flush-conntrack zone=5 \ 'ct_nw_src=10.1.1.2,ct_nw_dst=10.1.1.1,ct_nw_proto=17,ct_tp_src=2,ct_tp_dst=1' $ ovs-appctl dpctl/flush-conntrack zone=5 \ 'ct_nw_src=10.1.1.2,ct_nw_dst=10.1.1.1,ct_nw_proto=17,ct_tp_src=2,ct_tp_dst=1'
2018/06/11: "I suggest using ofproto/trace to figure out what's going on. See ovs-vswitchd(8) if you're not already familiar with it."
2018/06/11: "You can see the history of the configuration database by running "ovsdb-tool -mm show-log" on it. This might reveal what is happening, too."
"ovs-vsctl --may-exist add-port {{ info.ovs_bridge }} {{ interface }} {{ info.ovs_options }} -- set interface {{ interface }} type=internal"
2018/11/25 - ovs-testcontroller
pidof
I encountered a new command today. A simple one really. Just finds the pid (program id) of a running process by name. The pid can then be used as a filter in other commands. For example:
lsof -p $(pidof thunderbird) | grep -i tcp
LXC Container Research Items
- Memory inside Linux containers: unanswered questions about how to check memory utilization within a container rather than the machine
- LXC 1.0: GUI in Containers by Stephane Graber, notes about non root containers, and running skype in a container.
Tuesday, October 24. 2017
High Availability Calculations
From Corosync Cluster Engine: Designing High Availability comes a basic high availability calculation:
A = MTBF --------------- MTBF + MTTR MTBF = mean time between failures MTTR = mean time to repair A = probability system will provide service at a random time (ranging from 0 to 1)
"High Availability is achieved through the manipulation of MTBF and MTTR parameters of system design to meet availability requirements. "
Sunday, October 22. 2017
ESXi vSwitch Load Balancing Details
Matt Oswalt has an article called ESXi vSwitch Load Balancing Woes where he deep dives into VMware's ESXi virtual switch and, based upon personal experience, discusses port channels, hash algorithm selection, and path selection. I need to refer back to this to see if networks I've seen conform to what he discusses.
Listing Package Dependencies
From How do I find the build dependencies of a package?, there are some commands for finding build package dependencies:
- full list of dependencies: apt-rdepends --build-depends openoffice.org
- dependencies not installed: apt-rdepends --build-depends --print-state openoffice.org | grep NotInstalled
- dependencies for top most package: apt-rdepends --build-depends --follow=DEPENDS
- show control file for source: apt-cache showsrc
| grep ^Build-Depends - show control file for source: apt-cache showsrc
GCHQ: Email security and anti-spoofing
A document of suggestions to help make email more secure:
Email security and anti-spoofing
Talks about TLS, DKIM, DMARC and SPF with a few barebone configuration examples.
Friday, October 20. 2017
Ramblings for a Friday (Reading the Communications of the ACM)
How many subscribe to scholarly things like 'Communications of the ACM'? or the 'IEEE Spectrum'?
I then I took it one step further and sat at a table by myself down at the club, and read it cover to cover. With a beer or two.
And something that hasn't happened in a long time, or well, ever, I got called out as a sterotype: looking like a geek from San Francisco. Well, that was what the woman, who also had no conversational partner, used to strike up a conversation.
Anyway, back to the magazine, who would have thought, that in the CACM, you see such interesting things like:
"Divination is the practice of occultic ritual as an aid in decision making", on page 7 no less. Which goes on to say that "... divination truly allows us to consult the divine, we can view it simply as a form of randomization, which is recognized as a powerful construct in game theory, and algorithm design." ... "randomization is a powerful way to deal with incomplete information" ... "they are simply randomizing in the face of uncertainty about rain, pests, and more, but this randomization comes with a belief in the divine source of the decision".... "when the accept/reject decision pivots on issues such as significance and interestingness, which can be quite subjective".
Another article: "Assuring Software Quality by Preventing Neglect".... which is a 'grayhat' problem when compared to blackhats (hackers who deploy software as a weapon with malicious intent) and whitehats (setting safeguards against defective products)". With a few interesting conversation starters being, I think: ".... during maintenance cycles, they do not correct the old source code comments, seeing such edits as risky and presumptuous". ... "it is a failure of degree, a failure to pay enough attention and take enough trouble" ... which leads to a theory of "ethics of care which displaces the classical agent centered morality of duty and justice, enduring patient-centered morality as manifest real-time in relationships".
And, well, something which probably everyone knows, but has been evolving through a series of articles in the magazine, is that machine learning algorithms require training time via "generative adversarial networks", which means you need to introduce some sneaky garbage to improve the odds of appropriate machine learning and matching black boxes.
And another, in which I can personally associate: "Multitasking Without Thrashing". "Human context switching is more complicated than computer context switching. Whereas the computer context switch replaces a fixed number of bytes in a few CPU registers, the human has to recall what was 'on the mind' at the time of the switch, and, if the human was interrupted with no opportunity to choose a 'clean break', the human has to reconstruct lost short term memory" ... "if you have several important tasks, your brain can get stuck in a decision process that can take quite a long time to decide -- a situation known as the choice uncertainty problem". ... "thrashing happens to human multi-taskers when they have too many incomplete tasks. They fall into mood of 'overwhelm' in which they experience considerable stress" ..... "context switching is _not_ the cause of thrashing. ... the cause of thrashing is the failure to give every active task enough space for its working set" ... "when a task's working set is in your workspace, protect it from being unloaded as long as the task is active .. analog: protect working sets of active tasks and do not steal from other tasks". ... which leads to the rather obvious conclusion: "to exit the thrashing state, you need to reduce demand or increase your capacity".
Ok one more: "what we have instead is a society moving towards prosthetic brains that can be monitored at all times by the state, without the inconvenience of having to have everyone check in each day at the police station". When you read this, were you thinking "this isn't me! This isn't now!"? Well, even in present time, this is visible by what we write, where we write, what we read, where we read, who we interact with, what we interact with, ....
Thursday, October 19. 2017
Linux Network Performance Optimization
- Mark Wagner: performance tuning
- DPDK: How to get best performance with NICs on Intel platforms
- CloudFlare: How to achieve low latency with 10Gbps Ethernet
But then again, all this is probably old hat, and may have changed for 40gbps adapters. What about the Mellanox Connect-X 5 100 gbps adapters?
Wednesday, October 18. 2017
Debian Installer
Debian Installer is a complicated tool. I am trying to figure out where to start. Well, some starting pages:
- DebianInstaller: starting page for many things debian installer
- Debian-Installer: Building the installer yourself
- DebianInstaller Preseed: various ways to get preseed files to the debian installer
- Debian Testing Install
- Netboot: the daily release version
- Debian Installer Internals: read to understand what happens during the installation process
- Netbooting and Firmware: not absolutely required as there is a non-free cdimage debian repository from which boot .iso files can be obtained with non-free firmware.
To troubleshoot some recent Buster installation problems, I know now to use DEBCONF_DEBUG=5 in my boot files.
Debian Git Server, which includes debian installer packages.