Naprogramujte systém na analýzu Logov.

Here is a complete practical Python exercise designed for intermediate/advanced students. It includes:

• Context
• Dataset description
• Tasks (increasing difficulty)
• Advanced analytical questions
• Expected deliverables
• Bonus challenges

You can give this as a 3–6 hour lab or a take-home assignment.

---

🧪 Practical Exercise: Advanced Nginx Log Analysis with Python

🎯 Goals

You are given raw Nginx access logs. Your objectives are to:

1. Parse and clean log data
2. Perform traffic analysis
3. Detect anomalies
4. Identify performance bottlenecks
5. Extract security insights
6. Answer complex analytical questions using Python

You may use:

• re
• datetime
• collections
• pandas
• matplotlib
• numpy
• Any standard Python library

---

📂 Dataset

You are given an Nginx access log file:

access.log

The log format is the default combined log format:

127.0.0.1 - frank [10/Oct/2025:13:55:36 +0000] "GET /api/products HTTP/1.1" 200 2326 "https://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"

Fields:

Field	Description
IP	Client IP address
Remote user	Authenticated user (usually -)
Timestamp	Date and time
Method	GET/POST/etc
Path	Requested URL
Protocol	HTTP version
Status	HTTP response code
Size	Response size in bytes
Referrer	HTTP referrer
User-Agent	Client software

---

🧩 Part 1 — Parsing (Foundation)

Task 1.1

Write a parser that extracts the following fields:

• ip
• timestamp (as Python datetime)
• method
• path
• status (int)
• response_size (int)
• user_agent

Store the result as:

• A list of dictionaries OR
• A pandas DataFrame

---

Task 1.2

Compute:

• Total number of requests
• Number of unique IP addresses
• Number of unique endpoints
• Distribution of HTTP methods

---

📊 Part 2 — Traffic Analysis

Task 2.1 – Traffic Over Time

• Compute requests per minute
• Plot traffic time series
• Identify the peak traffic minute

---

Task 2.2 – Top Clients

Find:

• Top 10 IP addresses by request count
• Top 10 IPs by total bandwidth usage
• Top 10 requested endpoints

---

Task 2.3 – Status Code Analysis

• Count occurrences of each status code
• Compute percentage of 4xx errors
• Compute percentage of 5xx errors
• Find endpoints generating the most errors

---

🚨 Part 3 — Security Analysis

Task 3.1 – Suspicious IP Detection

Identify IPs that:

• Made more than 100 requests in 1 minute
• Generated more than 20% 4xx responses
• Accessed /admin, /wp-login, /phpmyadmin

Are any likely brute-force attempts?

---

Task 3.2 – User-Agent Analysis

• Detect possible bots
• Identify requests with empty or suspicious user agents
• Find the most common browsers

---

Task 3.3 – Path Exploration Attacks

Detect potential scanning behavior:

• Requests to many non-existing pages (404-heavy IPs)
• Sequential numeric ID access (e.g., /product/1, /product/2, /product/3…)

---

⚡ Part 4 — Performance Analysis

Task 4.1 – Slow Endpoints (Simulated)

If response time is available (or simulate it):

• Compute average response size per endpoint
• Identify unusually large responses
• Detect endpoints with abnormal patterns

---

Task 4.2 – Heavy Bandwidth Consumers

• Which IP consumed the most data?
• What percentage of total traffic did top 3 IPs consume?

---

🧠 Part 5 — Complex Analytical Questions

Answer the following:

---

❓ Q1

Is traffic evenly distributed throughout the day? If not:

• When are the peak hours?
• When is traffic minimal?

---

❓ Q2

Is there evidence of a DDoS attempt?

Hint:

• Look for traffic spikes
• Look for high request rates from single IPs
• Look for abnormal 4xx spikes

---

❓ Q3

Are 5xx errors correlated with high traffic periods?

Compute:

• Correlation coefficient between request volume and 5xx count per minute

---

❓ Q4

Which endpoints are most likely targets of malicious activity?

Support your answer with:

• Error rate
• Request bursts
• IP diversity

---

❓ Q5

Can you cluster IPs into behavioral categories?

For example:

• Normal users
• Bots
• Attackers
• Crawlers

(Use request frequency, error ratio, endpoint diversity.)

---

🏆 Bonus Challenges (Advanced)

🔹 1. Build a Log Analyzer Class

Create a reusable class:

class NginxLogAnalyzer:
    def __init__(self, log_file):
        ...

    def parse(self):
        ...

    def traffic_summary(self):
        ...

    def detect_anomalies(self):
        ...

---

🔹 2. Anomaly Detection (Statistical)

Use:

• Z-score
• IQR
• Rolling average

To detect abnormal traffic spikes.

---

🔹 3. Geolocation Analysis

Using an IP geolocation API (optional):

• Map traffic by country
• Identify suspicious geographic patterns

---

🔹 4. Real-Time Stream Processing

Simulate tailing a log file and analyze:

tail -f access.log

Process logs incrementally.

---

📦 Deliverables

Students must submit:

1. Python script or Jupyter notebook
2. Plots (traffic, errors, etc.)
3. Written answers to analytical questions
4. Explanation of detected anomalies
5. Code quality (structure, modularity)

---

📈 Evaluation Criteria

Criteria	Weight
Correct parsing	15%
Traffic analysis	20%
Security insights	20%
Advanced analytics	20%
Code quality	15%
Visualization quality	10%

---

🎓 Difficulty Level

• Parsing → Easy
• Aggregation → Medium
• Security detection → Medium+
• Correlation & anomaly detection → Hard
• Behavioral clustering → Advanced

Previous Post Next Post