Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix missing events from httpfilter #136

Merged
merged 3 commits into from
Jun 8, 2023

Conversation

grcevski
Copy link
Contributor

@grcevski grcevski commented Jun 7, 2023

While working on the integration tests for Rust I noticed a strange edge case, if the Rust server wasn't sending any data back on an API call, like our '/smoke' API, we weren't catching the smoke events. In my tests I had to add a fake plaintext response to make the tests pass.

After digging further I found out that the actix web framework has an optimisation, if you send multiple identical requests that don't have any response, it reuses the last accepted connection, so technically it doesn't call the accept4 syscall, nor it allocates a new socket, if the connection is not closed.

Since sometimes we might miss the first accept call based on timing, we weren't capturing the /smoke API calls, since we never found a filtered connection info. To mitigate this I added a kprobe on tcp_rcv_established which will establish the filtered connection info if there's not one already. Accept and connect are allowed to overwrite this info, so technically it's there only if we never saw an accept happen. I don't think this can happen with a client call, but I'll experiment more and see if I have to worry about client calls and matching what kind of request is established. Right now if it's missing we assume a server request.

I also reworked some of the docker compose files for Java and Rust. Naming things as rusttestserver was problematic when we use make cleanup-integration-test which I had to do a lot of :). The cleanup uses the regular docker-compose.yaml, and if the names don't match we don't bring down all of the containers.

@grcevski grcevski added bug test Issues related to testing labels Jun 7, 2023
@grcevski grcevski requested a review from mariomac as a code owner June 7, 2023 19:01
ENTRYPOINT ["/greetings"]
#CMD [ "strace", "-f", "/greetings" ]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this commented out for now because it's handy for debugging the tests with the kprobes.

@@ -15,7 +14,7 @@ async fn greeting(item: web::Json<MyObj>) -> HttpResponse {
}

async fn smoke() -> HttpResponse {
HttpResponse::Ok().content_type(ContentType::plaintext()).body("hello")
HttpResponse::Ok().into()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed any content reporting, just send empty 200 OK response.

network_mode: "service:rusttestserver"
pid: "host"
network_mode: "service:testserver"
pid: "service:testserver"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had this to "host" by mistake, we should try harder :).

@codecov-commenter
Copy link

Codecov Report

Merging #136 (26546db) into main (40e14a6) will increase coverage by 26.45%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##             main     #136       +/-   ##
===========================================
+ Coverage   40.36%   66.82%   +26.45%     
===========================================
  Files          30       30               
  Lines        2056     2107       +51     
===========================================
+ Hits          830     1408      +578     
+ Misses       1169      584      -585     
- Partials       57      115       +58     
Flag Coverage Δ
integration-test 66.82% <100.00%> (?)
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/ebpf/httpfltr/httpfltr.go 78.06% <100.00%> (+50.91%) ⬆️

... and 28 files with indirect coverage changes

@grcevski grcevski merged commit 9e9982c into grafana:main Jun 8, 2023
@grcevski grcevski deleted the fix_filter_missing branch June 8, 2023 14:13
@grcevski
Copy link
Contributor Author

grcevski commented Jun 8, 2023

Thanks Mario!

mattdurham pushed a commit to mattdurham/beyla that referenced this pull request Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Issues related to testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants