euLISA
Discussion 2019.08.13
Best practices
Sizing considerations
1 appliance
The example: 16 cores and 32G RAM for 15k fps
15k fps max if ADS is present
Storage
For 1-6T: relatively cheap to even put on SSD/NVMe
Sequential throughput
Latency
IOPS
CPU/RAM
dashboards and widgets
Calculated in the background
Views that report on particular views
channels
Or anything we specifiy, e.g. 'port 80' or 'src net 10.0.0.0/0'
Either 1 source
profiles
Consists of channels
Basis for reporting and dashboarding
Preselected data
Faster analytics
Subset of data
flows/sec
bandwidth and flows
Correlation is weak
Correlate in a way that more results in more
Use of 500G store
If only NetFlow v9 depth, half the size is needed
2 weeks for 15 fps peak network, full IPFIX
10k users
typicals
duplications in flow collection do occur
IT-heavy customers go up to 10 fps per user
200 fpm (~4 fps) per user for office
anywhere between 10k fps to 200k fps
1 flow = 1 log
Proliferation of views
Complexity
Application use per city/country/region/total
Dashboards, reports, alerts will grow in numbers
Engineers need microscopes, managers need overviews
Retention times
Flow forwarding
Point in time
Quotas
Data storage of profiles
Analysis (optional)
An example of partitioning
O365
NPM statistics
https
Top views, show in time, table views
Data we collect
Flowmon specifics
IPFIX extensions
Basics
Preface and agenda
Let us get to know a large customer
Slides about typical use cases
Presentation of theory
NPMD module
Satellite views
All-Company views
Regions
Countries
Cityscape aerial views
Location, city, country views
Use of particular applications
Microscope granularity
Troubleshooting, deep understanding
Time series of usage
Flow lists with detailed information
1 particular IP or user
Discussion 2019.08.09
Rescheduled
Discussion 2019.07.05
Discussed
What can we do regarding other protocols?
HTTPS
Quota management
Subdivision of applications
Subordination
Grouping
Roles
admin, read-only, analyst
Logging
user activity logs
system messages
Custom extractors
5 of them total per system
e.g. HTTP cookies
e.g. SOAP/XML headers
APM basics
Basic depths
Not the data itself: not HTTP content or actual SQL data
SQL: queries
HTTP: headers
SLA's
APM specifics
Resource use notes
Disk space
CPU performance
Possible XML examples
Show custom extraction
Deep dive into aggregated and custom stats
One example: application subcategories
Content in transactions
Fields
Show DBMS and HTTP basic depth, fields
SLA
Basic overview
APM
Look at HTTP and DBMS transactionally
ADS
Anomaly detection, monitor patterns in resource usage
NPMD
Resource usage ant-farm
Visibility based on IPFIX
Main topic: discussion of content parsing in APM, SOAP/XML
Discussion 2019.04.08
Cabling to be done at end of May
Risks (minor)
Context/feed
Which ip ranges are what
What applications
Interop
Training dates
On-demand web sessions
Brno training in Czech Republic
29.04.2019-30.4.2019
Discussion 2019.04.03
2nd phase clearances
timeframe
project name
Education, engineering team
Questions
preinstall
services.flowmon.com / disk
onsite sw upgrade
devices come installed
3. test scenarios
npmd
OOT
SRT
RTT
http/dbms - apm
anomaly inspection
dashboarding
2. link speed
1. Mgmt IP, mgmt conn
Dates, sites
Work to start first half of June
Datacenter flooring needs reinforcing
Discussion 2019.01.28
NDA
Everis NDA to be signed
Clearance
Now - France not mandatory
Future - Austria mandatory
Flowmon onsite
PROD
PrePROD
5 days tuning and on-the-job training
2 days hw
IP addresses
Management ports
Documentation to be prepared in advance
Documentation must be prepared in 2-3wks into feb
LLD
Authorization for TAP install
Port level
HLD
Valid location of rack within the datacenter, same rack install
Installation to start last week of february
Flowmon to deliver by feb 18
IXIA to deliver by feb 15
Discussion 2019.01.24
Delivery and dates
Installation to start 1 week after delivery
Follow up on exact delivery dates, pref feb.20 week
Support
No plans to provide VPN access
Testing
Important to know exit/acceptance criteria in advance
acceptance testing between PreProd and Prod install
Interaction to follow with euLISA test manager
performance
not discussed yet
redundancy
DoA
Share install docs with George
Question to address later: training in BRNO
SITES
Strasbourg, FR
single site, one datacenter
Production, preproduction in same rack
Details
Cabling requirements
Rack locations
FLOWMON
PANDUIT
cabling
Racks
IXIA
For PS quotation
QA, verification of customer expectations
Flowmon does not directly engage customer
Design
LLD depth limited until tuning is finished
Need to know: adaptive baselining is typically done on live traffic
Installation of modules
Training on site
Partner/customer education
On-site hardware installation
Production follows
Flowmon is present to support
Start installation in Strasbourg on PreProd
Flowmon leads
Legal
Required clearances
For facility access
To work on the project
NDA's to sign
In France, we will work with a partner who has clearance
By individuals
By the company
Integration (generic)
Traffic Recorder install and set-up
ADS driven
DDoS defender install and education
Definition of actions, if any
Learn baselines
Definition of protected address ranges
APM install and education
Definition of SLA's
Definition of SQL servers
Oracle, MS SQL, MySQL, PostgreSQL
Definition of web servers
Private keys for HTTPS
ADS tuning and education
Context
What IP is what
Tuning of methods
Thresholds
Filters
Basic network services, own address ranges
Collector/NPMD install and education
PDF reporting
Dashboarding
Realtime searching
Profiling
Optional: other netflow sources
Mirror traffic for Probes
Packet broker
Switching
Software modules installation
Installation of hardware
Flowmon components
DDoS Defender module
Mitigation capabilities
Scrubbing devices (a10, f5, radware)
Routing interaction
Detects volumetric attacks
Traffic Recorder module
Records traffic
Manual, on-demand
ADS event driven
APM module
Monitors HTTP and SQL transactions, shows transaction times and aligns those with defined SLA's (~'expectations')
Resides both on Collector and Probe
Software module
ADS module
Security and operational aspects
Learns and is taught normal operation, alerts on anomalies
Resides on Collector
Software module, resides on Collector
Probe
normally receives raw data
IPFIX/netflow export is via management interface
standard TCP or UDP traffic
enrich it
APM: mandatoy
netflow/IPFIX
10000 has 1x 10 Gbps SPF+
20000 has 2x 10Gbps SFP+
LC cable terminations (optics)
10GBASE-SR 850nm
Traffic is passed through
Packet broker (IXIA?)
Mirroring
Collector
2x1 Gbps mgmt RJ45 each
Runs modules, controls Probes
Collects and stores flow data
Hardware unit, rack server
