Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

json to hugo pages #19

Merged
merged 24 commits into from
Aug 12, 2022
Merged

json to hugo pages #19

merged 24 commits into from
Aug 12, 2022

Conversation

elreydetoda
Copy link
Collaborator

@elreydetoda elreydetoda commented Aug 3, 2022 •
edited
Loading

Part of addressing this: JupiterBroadcasting/jupiterbroadcasting.com#110

Alright, everything's working now 😁

NOTE: This can't be merged till I make the appropriate changes to utilize the new logic, or else it'll break the update flow that we've got with the GH action.

Summary of changes:

  • this actually could close Feat/adding pipenv #12 instead of that PR, I can remove those commits if we want or just add the requirements.txt back till we figure out Feat/adding pipenv #12
  • I've added 2 more packages python-frontmatter & pydantic-yaml
  • adding more typing to help validate/ensure some of the information we're ingesting/consuming from various locations (even the config)
    • part of this was defining what chapters look like per the spec
    • this also caused a lot of dictionaries to not be addressable as objects i.e. example_data['foo'] -> example_data.foo
  • migrated people & sponsors objects which were writing to data/ to now use content/ and writing as .md instead of .json
    • I used yaml for the frontmatter, because the output for json & toml isn't as well supported in the python-frontmatter library, and with pydantic-yaml doing a lot of the heavy lifting for ease of conversion, it just works 😅
  • documenting code more with docstrings
  • added some more checks if there were unexpected info, and logging errors if they are
  • simplifying show_exceptions for the jb_populate_episodes_urls function

@elreydetoda
adding pipenv file based on requirements.txt + gh action python verison
567928c
@elreydetoda
making modifications to docker stuff to use pipenv
9ba798b
@elreydetoda
updating README to use pipenv workflow
d94d183
@elreydetoda
making file executable
a5c8878
@elreydetoda
adding vagrantfile to let people use a pre-configured VM
e55ea0f
@elreydetoda
adding dev dependencies for linting
697a5ea
@elreydetoda
- created config validator (help verify format of config file)
4c59327 
- completely updated to use config validator new format (attrib instead of dict)
- documented jb_get_last_page_of_show
@elreydetoda
saving before formatting
2fd2b6a
@elreydetoda
documented jb_populate_episodes_urls
b8cdcff
@elreydetoda
- added model for JB_DATA's episode records
5b8afe3 
- documenting scrape_data_from_jb + sub-functions
- handling JupiterBroadcasting#16 by logging it till better solution
@elreydetoda
- adding better logging info when error
d1ce2ae 
- making empty Jbd_Episode_Record, so it can write episode out still
- added TODO for error talked about on GH
@elreydetoda
adding better logging if link is trying to get set on dataclass
7f755b1
@elreydetoda
- validating data recieved from fireside
9ee70fb 
- updating references with necessary changes for models
@elreydetoda
fixing newly found error from torrent on some pages
e2d11e5
@elreydetoda
removing uncessary import
e3d1b3a
@elreydetoda
- adding more parameter typing to parse_guests_in_ep
c700cf6 
- adding warning when there won't be direct downloads
- simplfying replace with raw string
@elreydetoda
- extending episode typing to include chapters
b06815b 
- adding hd_torrent to Jbd_eipsode_record
- renaming Jbd_episode_record to Jb_episode_record
@elreydetoda
adding clearer typing to episode model
0e28e2d
@elreydetoda
- changed logging numbers -> constants
cc31c46 
- now returning object instead of dict for get_podcast_chapters
- using html2text instead of html2text.html2text
@elreydetoda elreydetoda mentioned this pull request Aug 3, 2022
Open
7 tasks
@elreydetoda elreydetoda marked this pull request as ready for review August 3, 2022 12:34
@elreydetoda elreydetoda changed the title Feat/json to hugo pages json to hugo pages Aug 4, 2022
@elreydetoda elreydetoda mentioned this pull request Aug 4, 2022
Closed
@kbondarev
Copy link
Collaborator

@elreydetoda Good work, I managed to set it all up with pipenv and all.

One thing I'm not sure about:
Side effect of you using the UUID type for guid is that all of them were converted to be lowercase. It is most likely fine, but I can't find anywhere on podcast index whether it matters or not - is the guid case sensitive or not? (I assume not, but I kinda want to confirm that somehow)

@elreydetoda
Copy link
Collaborator Author

@elreydetoda Good work, I managed to set it all up with pipenv and all.

Yay 🎉 , I'm glad that you got everything working and tested 😁 (some day you'll have to tell me how you got it to work with your python setup 😅 )

BTW, I just pushed the last commit which addressed your other concern from the other repo, my bad for not pushing it earlier...
JupiterBroadcasting/jupiterbroadcasting.com#252 (comment)

@elreydetoda
Copy link
Collaborator Author

One thing I'm not sure about:
Side effect of you using the UUID type for guid is that all of them were converted to be lowercase. It is most likely fine, but I can't find anywhere on podcast index whether it matters or not - is the guid case sensitive or not? (I assume not, but I kinda want to confirm that somehow)

Interesting... 🤔 , it's probably happening with the conversion in the pydantic_yaml library...let me see if there's anything I can pass to modify that to all uppercase.

@kbondarev
Copy link
Collaborator

Yay tada , I'm glad that you got everything working and tested grin (some day you'll have to tell me how you got it to work with your python setup sweat_smile )

I installed a fresh Fedora 36 on a second SSD on my machine which I use only for testing distros or weird thing like that. And it all worked there out of the box 🙈

@elreydetoda
Copy link
Collaborator Author

Lol, we'll I'm just happy it worked 😁

@kbondarev kbondarev merged commit 6835cc6 into JupiterBroadcasting:main Aug 12, 2022
This was referenced Aug 12, 2022
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@elreydetoda @kbondarev