Entries by Raymond P. Burkholder

In looking at the output of 'lsof' for a process, I saw a couple of attached /proc entries. Looking at each, provides an interesting view into the process.

$ cat /proc/<pid>/statm
10593951 17514 5924 24288 0 82658 0
$ cat /proc/<pid>/status
Name:   code
Umask:  0022
State:  S (sleeping)
Tgid:   227783
Ngid:   0
Pid:    227783
PPid:   227577
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000
... more stuff ...

When using Cern's ROOT Data Analysis Framework, there are notes about possible compile time enhancements. It is best to take a look on which cpu variant the compiler will generate code, and for which target cpu the code will run. They need to be compatible.

AVX-512 is an interesting Wikipedia page describing the various cpu variant incarnations and their specialized instruction subsets.

Here are a few command line examples for examining what the gcc compiler sees as being available.

$ gcc -march=native -Q --help=target | grep march
  -march=                               skylake
  Known valid arguments for -march= option:

$ echo | gcc -dM -E - -march=native
  ... large quantity of flags ...

gcc x86 cpu options cmake command:

add_definitions(-march=native)

Even though this optimizes instruction use, it has to be used carefully. It will probably cause side effects with compiling code on one version of cpu and when copying the executables to a different machine, with possibly a different cpu variant.

Reference: How to see which flags -march=native will activate?

• H3 is a geospatial indexing system that partitions the world into hexagonal cells.
• Linux Performance Analysis in 60,000 Milliseconds - uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar, top, ...
• FOSDEM 22 - is a rather busy conference. This edition features 654 speakers, 731 events, and 100 tracks. We do our best to provide you with as much information and navigation options about the schedule as we can.
• Great Power is your guide to the emerging interstate rivalries which are less about brute force than asymmetric forms of conflict like cyber, information, and economic warfare.
• Nick Russo's Useful links - Technology, Lean Management, Theory of Constraints, Leadership, Business & Money, History, Life & Cognition,

• Does One Need DHCP(v6)?
• Is RFC 6939 Support Finally Here – Checking the Implementation of the “Client Link Layer Address Option” in DHCPv6

Tutorial: When to Write Which Special Member - "When explaining someone the rules behind the special member functions and when you need to write which one, there is this diagram that is always brought up. I don’t think the diagram is particularly useful for that, however."

Tutorial: Managing Compiler Warnings with CMake - " But how do you manage the very compiler-specific flags in CMake? How do you prevent your header files from leaking warnings into other projects?"

From the description of a talk 'March 16 Talk, "Deep Learning for Sequences in Quantitative Finance" with David Kriegman of UCSD and Two Sigma', an excellent summary:

The quantitative investment process can be viewed as one that takes in raw data at one end and executes trades that buy and sell financial instruments at the other end. The process naturally decomposes into steps of feature extraction, forecasting the returns of individual instruments, portfolio allocation to decide quantities to trade, and trading execution. Many of the steps in this process are readily expressed as machine learning problems that can be addressed using deep learning sequence methods.

For future reference:

When writing a about DNSSEC, I need to reference the RFC. But it's three RFCs (4033, 4034, and 4035), and possibly another (6840). It would be awfully nice to refer to "DNSSEC" with a single reference like "BCP 250".

Negative Trust Anchors - https://datatracker.ietf.org/doc/rfc7646/

In case of DNSSEC validation failures, don't switch resolvers - https://datatracker.ietf.org/doc/draft-livingood-dnsop-dont-switch-resolvers/

Added 2022/04/11

BIND 9.16 has dnssec-policy that makes algorithm rollover much easier. I recommend you start using that. Read more on migrating to dnssec-policy here: DNSSEC Key and Signing Policy

Additional links:

• bind9 DNSSEC Guide

Added 2022/05/05 - DNSSEC algorithm rollover HOWTO - some notes on how to upgrade a zone's DNSSEC algorithm using BIND.

Extracts from a mailop email list:

• little known control panel, for creating such email/web servers: Aetolos - secure open source control panel for virtual hosting
• Roundcube is the default webmail client these days, I am not sure if its a "replacement" for gmail, but it is very efficient and works great on mobile devices
• Postfix+dovecot is the most common combination for running your own services, once combined they offer pretty much all the features most people need (SMTP/IMAP/POP3/etc)
• ClamAV anti-virus is said to require at least 4GB of ram to load its default data set.
• For a full featured modern package with reasonable spam resistance I can vouch for Mailu which requires Docker as a basis. Under the hood, this is postfix as MTA, dovecot for IMAP/POP3, rspamd as anti-spam solution (I think the SPF/DKIM/DMARC stuff is located there, too), Roundcube or RainLoop as webmail (both pretty usable, probably not as feature-rich as GMail), PostgreSQL for account persistence, and REDIS as memory cache mostly for the rspamd engine. Supports Letsencrypt out of the box, of course. It does take some effort to set up the basics right (Docker and configuration) but then runs very reliably and is a breeze to manage. Sadly, it does not integrate Mailman, so I had to do that manually, which is kind of a pain.
• If you have a reasonable insight into unix things, you could do worse than going with OpenBSD and an OpenSMTPD setup along the lines of what Aaron Poffenberger describes in his tutorial here: opensmtpd for the real world mail server tutorial (originally a BSDCan session I believe).
• I’ve been running similar setups for years (but with exim as the MTA mainly for inertia reasons), documented mainly in articles you can find via the first link in my signature. The most comprehensive is perhaps Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools , others will be tagged with keywords like spam, mail, spamd, smtp and so forth.
• Gmail's webmail interface is one of the worst possible. It is very inefficient to operate, counter-intuitive, hides many important information from the user etc., not mentioning that it is simply ugly. There are many much nicer and more user-friendly webmails, like for example already mentioned Roundcube.
• Here I found an overview of various webmail apps, with screenshots of some. You can pick something that suits you: Top 15 Self-hosted open-source free web-based email clients
• I've begun with OpenSMTP, but I've switched to Postfix as the primary developer has stopped actively working on it. I now run Postfix, Dovecot, Solr (for the search), and Rspamd, and it's working perfectly!
• By the time you go through all the hassle of managing spam filters and getting your IP warmed up, Fastmail at $50/yr/mailbox looks pretty attractive.
• If you can find someone who resells Tucows' white label e-mail, they have a pretty good product for about $10/mailbox/yr for 5GB, $20 for 10GB, $30 for 15GB.
• take a look at mailcow dockerized. It's all you need (mail server, sogo groupware, antispam/-virus, 2fa,...). Put it on a VPS or your NAS and be happy.

MapiFi: Using Wi-Fi Signals to Map Home Devices

Imagine a map of your home with all of your connected devices (computers, TVs, voice control devices, printers, security cameras, etc.), in their location. You could then easily group devices into user-profiles, monitor Wi-Fi quality and activity in different areas of your home, and even locate a lost tablet in your home. MapiFi is a method to generate that map of the devices in a home. The first part of MapiFi involves the user (either a technician or the resident) walking around the home with a mobile device that listens to Wi-Fi radio channels. The mobile device detects Wi-Fi packets that come from all of the home's devices that connect to the gateway and measures their signal strengths (ignoring the content of the packets). The second part is an algorithm that uses all the signal-strength measurements to estimate the locations of all the devices in the home. Then, MapiFi visualizes the home's space as a coordinate system with devices marked as points in this space. A patent has been filed based on this technology. This paper was published in SCTE Technical Journal (see published paper at https://wagtail-prod-storage.s3.amazonaws.com/documents/SCTE_Technical_Journal_V1N3.pdf).

Log-based Anomaly Detection with Deep Learning: How Far Are We?

Software-intensive systems produce logs for troubleshooting purposes. Recently, many deep learning models have been proposed to automatically detect system anomalies based on log data. These models typically claim very high detection accuracy. For example, most models report an F-measure greater than 0.9 on the commonly-used HDFS dataset. To achieve a profound understanding of how far we are from solving the problem of log-based anomaly detection, in this paper, we conduct an in-depth analysis of five state-of-the-art deep learning-based models for detecting system anomalies on four public log datasets. Our experiments focus on several aspects of model evaluation, including training data selection, data grouping, class distribution, data noise, and early detection ability. Our results point out that all these aspects have significant impact on the evaluation, and that all the studied models do not always work well. The problem of log-based anomaly detection has not been solved yet. Based on our findings, we also suggest possible future work.

Reviewing source code from a security perspective has proven to be a difficult task. Indeed, previous research has shown that developers often miss even popular and easy-to-detect vulnerabilities during code review. Initial evidence suggests that a significant cause may lie in the reviewers' mental attitude and common practices. In this study, we investigate whether and how explicitly asking developers to focus on security during a code review affects the detection of vulnerabilities. Furthermore, we evaluate the effect of providing a security checklist to guide the security review. To this aim, we conduct an online experiment with 150 participants, of which 71% report to have three or more years of professional development experience. Our results show that simply asking reviewers to focus on security during the code review increases eight times the probability of vulnerability detection. The presence of a security checklist does not significantly improve the outcome further, even when the checklist is tailored to the change under review and the existing vulnerabilities in the change. These results provide evidence supporting the mental attitude hypothesis and call for further work on security checklists' effectiveness and design. Data and materials: https://doi.org/10.5281/zenodo.6026291

Pipeflow: An Efficient Task-Parallel Pipeline Programming Framework using Modern C++

Pipeline is a fundamental parallel programming pattern. Mainstream pipeline programming frameworks count on data abstractions to perform pipeline scheduling. This design is convenient for data-centric pipeline applications but inefficient for algorithms that only exploit task parallelism in pipeline. As a result, we introduce a new task-parallel pipeline programming framework called Pipeflow. Pipeflow does not design yet another data abstraction but focuses on the pipeline scheduling itself, enabling more efficient implementation of task-parallel pipeline algorithms than existing frameworks. We have evaluated Pipeflow on both micro-benchmarks and real-world applications. As an example, Pipeflow outperforms oneTBB 24% and 10% faster in a VLSI placement and a timing analysis workloads that adopt pipeline parallelism to speed up runtimes, respectively.

Software Engineering Meets Network Engineering: Conceptual Model for Events Monitoring and Logging

Abstraction applied in computer networking hides network details behind a well-defined representation by building a model that captures an essential aspect of the network system. Two current methods of representation are available, one based on graph theory, where a network node is reduced to a point in a graph, and the other the use of non-methodological iconic depictions such as human heads, walls, towers or computer racks. In this paper, we adopt an abstract representation methodology, the thinging machine (TM), proposed in software engineering to model computer networks. TM defines a single coherent network architecture and topology that is constituted from only five generic actions with two types of arrows. Without loss of generality, this paper applies TM to model the area of network monitoring in packet-mode transmission. Complex network documents are difficult to maintain and are not guaranteed to mirror actual situations. Network monitoring is constant monitoring for and alerting of malfunctions, failures, stoppages or suspicious activities in a network system. Current monitoring systems are built on ad hoc descriptions that lack systemization. The TM model of monitoring presents a theoretical foundation integrated with events and behavior descriptions. To investigate TM modeling s feasibility, we apply it to an existing computer network in a Kuwaiti enterprise to create an integrated network system that includes hardware, software and communication facilities. The final specifications point to TM modeling s viability in the computer networking field.

Early Detection of Network Attacks Using Deep Learning

The Internet has become a prime subject to security attacks and intrusions by attackers. These attacks can lead to system malfunction, network breakdown, data corruption or theft. A network intrusion detection system (IDS) is a tool used for identifying unauthorized and malicious behavior by observing the network traffic. State-of-the-art intrusion detection systems are designed to detect an attack by inspecting the complete information about the attack. This means that an IDS would only be able to detect an attack after it has been executed on the system under attack and might have caused damage to the system. In this paper, we propose an end-to-end early intrusion detection system to prevent network attacks before they could cause any more damage to the system under attack while preventing unforeseen downtime and interruption. We employ a deep neural network-based classifier for attack identification. The network is trained in a supervised manner to extract relevant features from raw network traffic data instead of relying on a manual feature selection process used in most related approaches. Further, we introduce a new metric, called earliness, to evaluate how early our proposed approach detects attacks. We have empirically evaluated our approach on the CICIDS2017 dataset. The results show that our approach performed well and attained an overall 0.803 balanced accuracy.

Cuckoo Trie: Exploiting Memory-Level Parallelism for Efficient DRAM Indexing

We present the Cuckoo Trie, a fast, memory-efficient ordered index structure. The Cuckoo Trie is designed to have memory-level parallelism -- which a modern out-of-order processor can exploit to execute DRAM accesses in parallel -- without sacrificing memory efficiency. The Cuckoo Trie thus breaks a fundamental performance barrier faced by current indexes, whose bottleneck is a series of dependent pointer-chasing DRAM accesses -- e.g., traversing a search tree path -- which the processor cannot parallelize. Our evaluation shows that the Cuckoo Trie outperforms state-of-the-art-indexes by up to 20%--360% on a variety of datasets and workloads, typically with a smaller or comparable memory footprint.

DeepScalper: A Risk-Aware Deep Reinforcement Learning Framework for Intraday Trading with Micro-level Market Embedding

Reinforcement learning (RL) techniques have shown great success in quantitative investment tasks, such as portfolio management and algorithmic trading. Especially, intraday trading is one of the most profitable and risky tasks because of the intraday behaviors of the financial market that reflect billions of rapidly fluctuating values. However, it is hard to apply existing RL methods to intraday trading due to the following three limitations: 1) overlooking micro-level market information (e.g., limit order book); 2) only focusing on local price fluctuation and failing to capture the overall trend of the whole trading day; 3) neglecting the impact of market risk. To tackle these limitations, we propose DeepScalper, a deep reinforcement learning framework for intraday trading. Specifically, we adopt an encoder-decoder architecture to learn robust market embedding incorporating both macro-level and micro-level market information. Moreover, a novel hindsight reward function is designed to provide the agent a long-term horizon for capturing the overall price trend. In addition, we propose a risk-aware auxiliary task by predicting future volatility, which helps the agent take market risk into consideration while maximizing profit. Finally, extensive experiments on two stock index futures and four treasury bond futures demonstrate that DeepScalper achieves significant improvement against many state-of-the-art approaches.

Effective Anomaly Detection in Smart Home by Integrating Event Time Intervals

Smart home IoT systems and devices are susceptible to attacks and malfunctions. As a result, users' concerns about their security and safety issues arise along with the prevalence of smart home deployments. In a smart home, various anomalies (such as fire or flooding) could happen, due to cyber attacks, device malfunctions, or human mistakes. These concerns motivate researchers to propose various anomaly detection approaches. Existing works on smart home anomaly detection focus on checking the sequence of IoT devices' events but leave out the temporal information of events. This limitation prevents them to detect anomalies that cause delay rather than missing/injecting events. To fill this gap, in this paper, we propose a novel anomaly detection method that takes the inter-event intervals into consideration. We propose an innovative metric to quantify the temporal similarity between two event sequences. We design a mechanism to learn the temporal patterns of event sequences of common daily activities. Delay-caused anomalies are detected by comparing the sequence with the learned patterns. We collect device events from a real-world testbed for training and testing. The experiment results show that our proposed method achieves accuracies of 93%, 88%, 89% for three daily activities.

ThorFI: A Novel Approach for Network Fault Injection as a Service

In this work, we present a novel fault injection solution (ThorFI) for virtual networks in cloud computing infrastructures. ThorFI is designed to provide non-intrusive fault injection capabilities for a cloud tenant, and to isolate injections from interfering with other tenants on the infrastructure. We present the solution in the context of the OpenStack cloud management platform, and release this implementation as open-source software. Finally, we present two relevant case studies of ThorFI, respectively in an NFV IMS and of a high-availability cloud application. The case studies show that ThorFI can enhance functional tests with fault injection, as in 4%-34% of the test cases the IMS is unable to handle faults; and that despite redundancy in virtual networks, faults in one virtual network segment can propagate to other segments, and can affect the throughput and response time of the cloud application as a whole, by about 3 times in the worst case.

Analyzing Enterprise DNS Traffic to Classify Assets and Track Cyber-Health

The Domain Name System (DNS) is a critical service that enables domain names to be converted to IP addresses (or vice versa); consequently, it is generally permitted through enterprise security systems (e.g., firewalls) with little restriction. This has exposed organizational networks to DDoS, exfiltration, and reflection attacks, inflicting significant financial and reputational damage. Large organizations with loosely federated IT departments (e.g., Universities and Research Institutes) often do not even fully aware of all their DNS assets and vulnerabilities, let alone the attack surface they expose to the outside world. In this paper, we address the "DNS blind spot" by developing methods to passively analyze live DNS traffic, identify organizational DNS assets, and monitor their health on a continuous basis. Our contributions are threefold. First, we perform a comprehensive analysis of all DNS traffic in two large organizations (a University Campus and a Government Research Institute) for over a month, and identify key behavioral profiles for various asset types such as recursive resolvers, authoritative name servers, and mixed DNS servers. Second, we develop an unsupervised clustering method that classifies enterprise DNS assets using the behavioral attributes identified, and demonstrate that our method successfully classifies over 100 DNS assets across the two organizations. Third, our method continuously tracks various health metrics across the organizational DNS assets and identifies several instances of improper configuration, data exfiltration, DDoS, and reflection attacks. We believe the passive analysis methods in this paper can help enterprises monitor organizational DNS health in an automated and risk-free manner.

ICT systems provide detailed information on computer network traffic. However, due to storage limitations, some of the information on past traffic is often only retained in an aggregated form. In this paper we show that Linear Gaussian State Space Models yield simple yet effective methods to make predictions based on time series at different aggregation levels. The models link coarse-grained and fine-grained time series to a single model that is able to provide fine-grained predictions. Our numerical experiments show up to 3.7 times improvement in expected mean absolute forecast error when forecasts are made using, instead of ignoring, additional coarse-grained observations. The forecasts are obtained in a Bayesian formulation of the model, which allows for provisioning of a traffic prediction service with highly informative priors obtained from coarse-grained historical data.

Without a specific functional context, non-functional requirements can only be approached as cross-cutting concerns and treated uniformly across all features of an application. This neglects, however, the heterogeneity of non-functional requirements that arises from stakeholder interests and the distinct functional scopes of software systems, which mutually influence how these non-functional requirements have to be satisfied. Earlier studies showed that the different types and objectives of non-functional requirements result in either vague or unbalanced specification of non-functional requirements. We propose a task analytic approach for eliciting and modeling user tasks to approach the stakeholders' pursued interests towards the software product. Stakeholder interests are structurally related to user tasks and each interest can be specified individually as a constraint of a specific user task. These constraints support DevOps teams with important guidance on how the interest of the stakeholder can be satisfied in the software lifecycle sufficiently. We propose a structured approach, intertwining task-oriented functional requirements with non-functional stakeholder interests to specify constraints on the level of user tasks. We also present results of a case study with domain experts, which reveals that our task modeling and interest-tailoring method increases the comprehensibility of non-functional requirements as well as their impact on the functional requirements, i.e., the users' tasks.

Software non-functional requirements address a multitude of objectives, expectations, and even liabilities that must be considered during development and operation. Typically, these non-functional requirements originate from different domains and their concrete scope, notion, and demarcation to functional requirements is often ambiguous. In this study we seek to categorize and analyze relevant work related to software engineering in a DevOps context in order to clarify the different focus areas, themes, and objectives underlying non-functional requirements and also to identify future research directions in this field. We conducted a systematic mapping study, including 142 selected primary studies, extracted the focus areas, and synthesized the themes and objectives of the described NFRs. In order to examine non-engineering-focused studies related to non-functional requirements in DevOps, we conducted a backward snowballing step and additionally included 17 primary studies. Our analysis revealed 7 recurrent focus areas and 41 themes that characterize NFRs in DevOps, along with typical objectives for these themes. Overall, the focus areas and themes of NFRs in DevOps are very diverse and reflect the different perspectives required to align software engineering with technical quality, business, compliance, and organizational considerations. The lack of methodological support for specifying, measuring, and evaluating fulfillment of these NFRs in DevOps-driven projects offers ample opportunities for future research in this field. Particularly, there is a need for empirically validated approaches for operationalizing non-engineering-focused objectives of software.

Found on a email list:

The Netgear GS108T is my typical go-to "not a dumb switch". 8 ports for about $80.

Make sure you get the v3 if you want most of the modern IPv6 L2 features (you also get some very limited L3 capabilities).

Extra bonus with the GS108Tv3, and anything else based on the RTL8380, is that you can run OpenWrt on it.

Security Orchestration, Automation, and Response Engine for Deployment of Behavioural Honeypots

Cyber Security is a critical topic for organizations with IT/OT networks as they are always susceptible to attack, whether insider or outsider. Since the cyber landscape is an ever-evolving scenario, one must keep upgrading its security systems to enhance the security of the infrastructure. Tools like Security Information and Event Management (SIEM), Endpoint Detection and Response (EDR), Threat Intelligence Platform (TIP), Information Technology Service Management (ITSM), along with other defensive techniques like Intrusion Detection System (IDS), Intrusion Protection System (IPS), and many others enhance the cyber security posture of the infrastructure. However, the proposed protection mechanisms have their limitations, they are insufficient to ensure security, and the attacker penetrates the network. Deception technology, along with Honeypots, provides a false sense of vulnerability in the target systems to the attackers. The attacker deceived reveals threat intel about their modus operandi. We have developed a Security Orchestration, Automation, and Response (SOAR) Engine that dynamically deploys custom honeypots inside the internal network infrastructure based on the attacker's behavior. The architecture is robust enough to support multiple VLANs connected to the system and used for orchestration. The presence of botnet traffic and DDOS attacks on the honeypots in the network is detected, along with a malware collection system. After being exposed to live traffic for four days, our engine dynamically orchestrated the honeypots 40 times, detected 7823 attacks, 965 DDOS attack packets, and three malicious samples. While our experiments with static honeypots show an average attacker engagement time of 102 seconds per instance, our SOAR Engine-based dynamic honeypots engage attackers on average 3148 seconds.